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Executive Summary 

The measurement of educational attainment has been a staple in social and 
economic research for decades, and federal data collections have long collected reliable, 
valid data on postsecondary academic degrees, including associate’s, bachelor’s, and 
graduate and professional degrees. This long-standing ability to measure degree-based 
educational credentials has played a significant role in helping researchers learn about the 
relationship between educational attainment and employment outcomes and in shaping 
public policy aimed at increasing Americans’ access to education. In recent years, the role 
of credentials other than academic degrees in helping out-of-school youth and adults obtain 
jobs and advance in careers has become a more prominent part of policy discussions. 

However, there is no federal data collection that measures the prevalence of 
industry-recognized certifications, state and local government issued licenses, 
subbaccalaureate educational certificates, noncredit instruction, and other nondegree 
credentials among the U.S. adult population. The number of U.S. adults who have these 
credentials and are using them in their jobs cannot be studied through institution-based data 
collection efforts, such as the National Center for Education Statistics’ (NCES) Integrated 
Postsecondary Education Data System (IPEDS). IPEDS collects information on certificates 
awarded, but is limited to counting awards based on credit-bearing instruction in institutions 
of higher education that participate in federal student financial aid programs. Students also 
receive educational certificates from institutions that do not participate in federal student 
financial aid programs, and they receive certifications and licenses from a wide variety of 
institutional bodies other than postsecondary institutions such as trade associations, private 
companies, state governments, and employers. As a result, no institutional sampling frame 
is available for enumerating these credentials at the national level. Additionally, information 
about certifications, licenses, and certificates is not adequately covered in federal data 
collections that survey adults directly. Without the ability to count certifications, licenses, 
and certificates, it is difficult to accurately assess their effects on social and economic 
outcomes and difficult to evaluate public policies designed to help Americans access and 
benefit from these credentials. 
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In 2009, a federal Interagency Working Group on Certificates and Certifications was 
formed to improve federal data collection about these credentials. In 2012, the working 
group’s scope expanded to include measures of participation and enrollment in federal 
surveys and the group is now referred to as the Interagency Working Group on Expanded 
Measures of Enrollment and Attainment (GEMEnA). GEMEnA consists of senior 
representatives from the Bureau of the Census, the Bureau of Labor Statistics (BLS), the 
National Science Foundation (NSF), as well as CEA, NCES, OMB, and OUS. The working 
group undertook the development of a short set of survey items to measure the prevalence 
of certifications, licenses, and educational certificates. This development effort culminated in 
the Adult Training and Education Survey (ATES) Pilot Study, a national household survey of 
noninstitutionalized adults ages 18 and over. 1 The pilot was conducted from September 
2010 to January 2011, and the results are presented in this summary. 

The ATES Pilot Study included measures of the prevalence of certifications, licenses 
and certificates. In addition, the study included measures of key factors that may be 
associated with these credentials, based on a literature review, expert input, and qualitative 
research with credential holders. These factors — measured by one or more items — included 
the level of effort to obtain the credential, examination and performance assessment 
requirements, type of provider or awarding body, industry recognition and occupational 
specificity of the credential, requirement for employment, suspension or revocation 
potential, and perceived labor-market value. Items representing key factors were used to 
describe credentials, and to analyze underreporting among convenience samples of adults 
known to have these credentials, based on information from the credential provider or 
awarding agency. Commonly reported credential characteristics (i.e., “defining 
characteristics”) of known credentials were then used to estimate overreporting among the 
convenience samples of known credential holders, and in the population. 

The research effort described in this summary was undertaken for questionnaire and 
procedural development purposes only. The information collected and published from this 
effort should not be used to generate or cite population estimates or other statistics because 
the sample design and data collection procedures for this pilot study were not intended for 
that purpose, but rather to support the evaluation of questionnaire items. 


1 Although these types of credentials are also relevant to out-of-school youth, the ATES Pilot Study was conducted 
with adults ages 18 or older. 
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Findings 

Before the ATES Pilot Study was conducted, cognitive research was undertaken to 
assess familiarity of terminology and adults’ understanding of potential survey items. The 
cognitive research included focus groups of known credential holders to learn the language 
that potential respondents use and in-depth, one-on-one interviews to get respondent 
feedback on potential questionnaire items. Key findings included: 

• First, focus groups and cognitive interviews indicated that the terms “certification” 
and “license” are not always distinguishable to respondents. Consequently, the 
decision was made to ask about certifications and licenses jointly. 

• Second, focus groups confirmed that certification and license holders do not view 
their certifications or licenses as educational credentials. Respondents instead 
considered these credentials to be professional qualifications. Consequently, 
questions about certifications and licenses were not asked within the questionnaire 
section on educational attainment. 

• Third, the cognitive interviews showed that respondents also had trouble 
distinguishing between the terms “certification” and “certificate.” The cognitive 
interviews revealed that some respondents who have a certification will answer that 
they have a “certificate” if the questionnaire asked about certificates before 
certifications. Therefore, the questionnaire was designed to specifically ask about 
certifications and licenses first and certificates second. 

• Finally, for certificates, a key finding was that respondents’ interpretation of the term 
“certificate” was varied when given little or no definitional context. Accordingly, a 
definition of certificate, as well as examples of certificate types and appropriate 
certificate providers were included in the lead item asking whether or not a 
respondent has a certificate. 

Based on the ATES Pilot Study, approximately 38 percent of adults reported holding 
a certification, license, or subbaccalaureate educational certificate in 2010-11, which 
translates into roughly 82 million adults in the United States. About 30 percent of adults 
reported holding a certification or license and 14 percent reported they earned a certificate. 
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These estimates are broadly comparable to estimates from other surveys of similar 
populations, including the nonfederal Princeton Data Improvement Initiative (RDM) for 
certifications/licenses, and the Census Bureau’s Survey of Income and Program 
Participation (SIPP) for certificates. 2 

Key factors for certifications and licenses were relatively consistent across these 
credentials, but the key factors for educational certificates showed a concerning level of 
variation. For certifications and licenses, most items examined showed the expected 
results — for example, 90 percent of certifications or licenses were pursued for mainly work- 
related reasons, and 91 percent of the most recent work-related certifications or licenses 
required respondents to pass a test or exam — however, certain types of items requiring 
detailed information were problematic. For example, it was difficult for respondents to report 
on the amount of time spent in coursework or training for the credential. Responses to key 
factor items for certificates were less predictable compared to certifications and licenses. 

For example, 26 percent of certificates were reported as having been obtained from “other” 
sources besides the types of educational institutions that most commonly offer a certificate 
program of study. The level of variation suggests that either the interpretation or definition of 
certificates is too broad to be adequately measured with the current approach. 

Measurement of certificates, therefore, requires further research. 

The study also included an evaluation of proxy reports; these were reports of 
credentials made by another adult member of the household on behalf of the sampled adult. 
The results of the analysis suggest that proxy respondents were a reasonable source of 
information on the main credential items and on some of the key factor items, such as 
whether the credential was obtained for work-related reasons, but not a good source of 
information on items that prompt respondents for detailed information (e.g., credential 
name, the year earned, or the time to complete a credential). 


2 The PDII data are available at http://irs.princeton.edu/Conferences/PDII/PDII RDD Survey. dta , and the SIPP 2008 
data are available at http://www.bls.census.gov/sipp ftp.html . 
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Recommendations 


The ATES Pilot Study findings have implications for determining a parsimonious set 
of items that could be included in other federal household surveys of adults — either in the 
near future for those items requiring no further development, or after additional testing has 
been conducted to address the measurement issues identified in the pilot. It is important to 
note that although results of the pilot are used in this section to recommend items for 
researchers with various goals, the properties of the items may differ in other survey 
administration contexts. 

The remainder of this section provides the recommendations of GEMEnA for items 
to use in other household surveys with the following research goals: 

• Counting individuals with a certification or license; 

• Distinguishing between certifications and licenses; 

• Describing the field of certification or license; and 

• Determining whether the certification or license is related to work. 

The recommendations of GEMEnA are based on the results of the ATES Pilot Study, 
described in this report, as well as GEMEnA’s subsequent survey development work 
conducted in preparation for a related study, the 2013 National Adult Training and 
Education Survey (NATES) Pilot Study. 3 

Counting individuals with a certification or license 

To count the number of individuals in the United States who have a certification or 
license, GEMEnA recommends pilot item CN1: 

Now I'd like to ask you about professional certification and licensure. Do 

you/person have a professional certification or a state or industry license? 


3 The subsequent survey development work consisted of 14 cognitive interviews conducted in preparation for the 
2013 NATES Pilot Study, as well as additional focus groups of certificate holders — many of whom also held 
certifications and licenses — conducted to inform future potential administrations of the NATES. The 2013 NATES Pilot 
Study is a household survey of adults being conducted to test the feasibility of administering an education and training 
survey by mail rather than by telephone. More information on the NATES is available on the GEMEnA website, at 
http://nces.ed.gov/survevs/qemena/strand4.asp . 
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This recommended item is referred to as the main certification/license item; it is an 
aggregate, broad measurement item that could be administered alone or in conjunction with 
credential-specific items to provide counts of adults with work-relevant credentials. To 
reduce measurement error, survey sponsors may consider adding to this item one to three 
of the items that measure key defining characteristics of certifications/licenses. These items 
include whether or not the certification/license is work-related (CN6); if the individual had to 
pass a test or exam (CNIOb), and; if the certification/license is transferable (CN15b). When 
considering these additional items, however, analysts should recognize their potential for 
eliminating true credentials if used as screening questions. Not all certifications and licenses 
require a test or exam, for example. 

In contrast, the main certificate item was found to have a high underreporting rate 
among the seeded sample respondents in the ATES Pilot Study and is not recommended 
for use at this time. The main certificate item and the series of items tested in the certificate 
questionnaire section require additional testing before they should be used. This additional 
testing is part of the ongoing development work of GEMEnA. 

Distinguishing between certifications and licenses 

Because certifications reflect a demonstration of skill while licenses convey a legal 
authority to work in an occupation, the questions researchers ask about these two types of 
credentials may differ. The ATES Pilot Study asked a follow-up question (item CN4) to the 
main certification/license question to see if the respondent considered a credential a 
certification, a license, or both. However, the focus group, cognitive interview, and ATES 
Pilot Study findings, as well as GEMEnA’s subsequent survey development work, confirmed 
that some respondents have difficulty distinguishing between certifications and licenses, 
particularly in occupations (including many healthcare fields) in which an industry- 
recognized certification demonstrating skill attainment is the main requirement for the state 
license. Therefore, GEMEnA recommends the inclusion of an item designed to help 
determine whether a respondent holds a certification or a license, rather than rely on 
respondents’ direct report using item CN4. 

One of the defining differences in the definitions of certifications and licenses in the 
ATES Pilot Study was the entity that awarded the credential. Certifications are awarded by 
industry or professional organizations while licenses are issued by government agencies. 



The ATES pilot study asked the respondent what kind of organization awarded the 
credential with item CN14, which asked respondents whether they were certified/licensed 
by their state, industry, a company, a professional association, or some other organization. 
However, based on the ATES findings and GEMEnA’s subsequent survey development 
work, GEMEnA recommends the following question be used instead of ATES item CN14 to 
distinguish between certifications and licenses: 

Who issued this certification or license? (Mark one.) 

□ Federal, state, or local government 

□ Professional or trade association (for example, Pediatric Nursing Certification 
Board, National Exercise and Sports Trainers Association, CompTIA) 

□ Business or company (for example, Microsoft, 3M Company, Xerox) 

□ Other group or organization (specify) 

Describing the field of certification or license 

GEMEnA recognizes that information about the field, industry, or occupation of a 
credential is critical to answering important questions about the relationship between 
employer needs and the supply of human capital. The dynamic of credential supply and 
demand takes place within specific occupational fields. National household studies 
measuring certifications and licenses should consider including questions to identify the 
field of the credential. The ATES Pilot Study asked respondents for verbatim responses to a 
question about the name of the certification or license (CN3A) and the kind of work it is for 
(CN3). Based on the ATES findings and GEMEnA’s subsequent survey development work, 
GEMEnA recommends the following questions be used instead of ATES items CN3A 
and CN3: 

What is the name of your [TEXT1 ][MOST RECENT] certification or license? 

Please do not use abbreviations. 

What kind of work is this certification or license for? (For example: teaching, 

vocational nursing, computer network administration, auditing, truck driving) 

Surveys with limited space can limit these questions to respondents’ most recent 
certification or license, while surveys with more space can collect information on multiple 
credentials of this type. The responses to these questions can be collected verbatim and 
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used individually, or used together to develop a set of occupation categories after data 
collection. The second question gives coders additional information to clarify the response. 

Determining if the certification or license is related to work 

GEMEnA has heard from the Council of Economic Advisors, prominent researchers, 
and other experts about the importance of specifying whether a credential is related to the 
respondent’s current job. The question used in the ATES Pilot Study (item CN6) focused on 
whether or not the credential was primarily “work-related” and was found to be associated 
with true certifications and licenses in the seeded sample. However, based on GEMEnA’s 
preference to focus the item on the respondent’s current job, the following item was tested 
in cognitive interviews and included in the 2013 NATES Pilot Study: 

Is this certification or license for the job you have now? If you are currently not 

employed, please answer “no.” (Mark one.) 

□ No 

□ Yes, and it is required for my job 

□ Yes, but it is NOT required for my job 

GEMEnA recommends this adaptation of the ATES question to clarify the 
relationship between the credential and the current job and to distinguish between 
credentials required for the job and those related to, but not required for, the job. Household 
surveys conducted by telephone or computer-assisted personal interviews may need to 
adapt this item for oral administration. 

Next Steps on Education, Training, and Credentials for Work 

With GEMEnA’s expert guidance and support, NCES has embarked upon a 
multifaceted effort to apply best-practice survey development principles towards the goal of 
developing valid national measures of the participation in and credentialing of education and 
training for work, and to build government-wide consensus for the adoption of these 
measures. 
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GEMEnA’s portfolio is described in Chapter 4 and includes four main strands 
of work: 

1. Develop and deploy a core set of survey items related to the prevalence and key 
characteristics of industry-recognized certifications and occupational licenses. 

2. Develop and deploy of a core set of survey items related to the prevalence and 
key characteristics of subbaccalaureate educational certificates. 

3. Consider new and revised measures of participation in education and training 
designed to prepare out-of-school youth and adults for work. 

4. Support NCES in the development of a new household study on education, 
training, and credentials for work. 

The work described in this pilot study is in support of strands 1 and 2 — to develop 
and deploy a core set of survey items related to the prevalence and key characteristics of 
industry-recognized certifications, occupational licenses, and subbaccalaureate educational 
certification. The results of this pilot study also inform strand 4 — to develop a new 
household study on education, training, and credentials for work. 
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Foreword 

In February of 2009, President Obama in his State of the Union address called for 
the U.S. to become once again “first in the world” in postsecondary educational attainment 
by the year 2020. As a means to this goal, he asked each American “...to commit to at least 
one year or more of higher education or career training. This can be a community college or 
a four-year school, vocational training or an apprenticeship. But whatever the training may 
be, every American will need to get more than a high school diploma.” Concurrently, the 
deepening recession was leading to greater interest in the role of industry-conferred and 
recognized credentials in preparing adults for specific, living-wage jobs. 

In this context, the Council of Economic Advisors and the Office of Management and 
Budget partnered with the Office of the Under Secretary in the U.S. Department of 
Education to convene senior staff from the Census Bureau, the Bureau of Labor Statistics 
(BLS), and the National Center for Education Statistics (NCES) to discuss ways to improve 
federal statistical data on educational certificates and industry-recognized certifications in 
order to meet research and policy data needs. After a series of initial meetings outlining the 
goals and resources of the project, an interagency working group was formed in December 
of 2009 to develop survey items on the prevalence and key characteristics of these 
credentials. Since its initial conception, the group’s composition has expanded to include 
the National Science Foundation and its charge has expanded to include enrollment and 
participation in education and training for work. Current and past members of this 
Interagency Working Group on Expanded Measures of Enrollment and Attainment 
(GEMEnA) are listed in appendix A. 

This report describes the rigorous process of survey item development GEMEnA 
undertook to meet its initial goal: the identification of a short set of survey items that would 
count the number of educational certificates and industry-recognized certifications in the 
U.S. adult population. The report includes detailed analyses of a pilot study designed to 
evaluate item validity by comparing its results against population estimates from other 
sources and by assessing the key characteristics of known credentials. The final chapter 
contains recommendations for using tested items and a discussion of next steps in 
development. These recommendations and their associated measurement discussions may 
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help survey developers decide how best to incorporate these items into surveys about 
education, training, and occupational credentials for out-of-school youth and adults. 

GEMEnA guided the item development work, providing detailed feedback at each 
stage, reviewing table results, discussing findings, and developing recommendations. 
GEMEnA’s members have also worked within their respective agencies to find opportunities 
to further develop, test, and use the validated questionnaire items from this project. NCES 
managed, staffed, and funded the work described in this report and is responsible for its 
contents. 

Jack Buckley 
Commissioner 

National Center for Education Statistics 
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1. Introduction 


Education and training beyond high school are important for securing opportunities 
for high-wage jobs in the United States. Academic degrees awarded by institutions of higher 
education represent a key component of the post-high-school credentials available to the 
American labor force. Other credentials, such as industry-recognized certifications, 
occupational licenses, and subbaccalaureate educational certificates have also emerged 
as key credentials with potential labor market value. This report describes work undertaken 
by the federal Interagency Working Group on Expanded Measures of Enrollment and 
Attainment (GEMEnA) to develop a short set of survey items to measure the prevalence 
of these credentials. 

The development of survey measures to enumerate adults with certifications, 
licenses, and certificates culminated in the Adult Training and Education Survey (ATES) 
Pilot Study, a national household survey of noninstitutionalized adults ages 18 and over. 

The primary objective of the study was to evaluate a set of survey items in order to 
determine the most parsimonious set of items needed to accurately measure the 
prevalence of certifications, licenses, and certificates in the U.S. adult population. 

The purpose of this report is to present the results of this evaluation and make 
recommendations for survey items to use in existing and future federal data collections. 

The research effort described in this report was undertaken for questionnaire and 
procedural development purposes only. The information collected and published from this 
effort should not be used to generate or cite population estimates or other statistics. 

1.1 Need for Data 

The link between educational attainment and a host of positive social and economic 
outcomes is well-documented (e.g., Bailey, Kienzl, and Marcotte 2004; Kane and Rouse 
1993; Pascarella and Terenzini 2005). The measurement of educational attainment has 
been a staple in social and economic research for decades, and federal data collections 
have long collected reliable, valid data on educational degrees, including associate’s, 
bachelor’s, and graduate and professional degrees. This long-standing ability to measure 
degree-based educational credentials has played a significant role in helping researchers 


1 



learn about the relationship between educational attainment and employment (and other) 
outcomes and in shaping public policy aimed at increasing Americans’ access to education. 

The relationship between the attainment of a postsecondary credential and job 
prospects has recently been prominent in the national conversation (Baker 2011; Bureau of 
Labor Statistics 2012; Career One Stop n.d.; Carnevale, Rose, and Hanson 2012; 
Carnevale, Smith, and Strohl 2010; Julian and Kominski 2011). This attention has brought 
to light certain weaknesses in our federal data collection systems, most notably the lack of 
information on nondegree credentials that have labor market value. These credentials 
include industry-recognized certifications and occupational licenses. They also include 
subbaccalaureate educational certificates that prepare adults for work. To understand how 
adults acquire work-related credentials, policymakers and researchers also need data on 
the enrollment and participation of out-of-school youth and adults in education and training. 
Such training may include noncredit instruction in community colleges, formal on-the-job 
training, and adult basic skills education. There is currently no federal data collection that 
measures the prevalence of industry-recognized certifications, occupational licenses, and 
educational certificates among the U.S. adult population. The number of U.S. adults who 
have these credentials and are using them in their jobs cannot be studies through 
institution-based data collection efforts, such as the National Center for Education Statistics’ 
(NCES) Integrated Postsecondary Education Data System (IPEDS). Although IPEDS 
collects information on certificate programs, it is limited to counting annual awards based on 
credit-bearing instruction in institutions of higher education that participate in federal student 
financial aid programs. Many out-of-school youth and adults receive educational certificates 
from institutions that do not participate in federal student financial aid programs, and many 
receive certifications and licenses from a wide variety of institutional bodies such as trade 
associations, private companies, state governments, and employers. As a result, no 
institutional sampling frame is available for enumerating these credentials at the national 
level. Additionally, information about certifications, licenses, and certificates is not 
adequately covered in federal data collections that survey adults directly. Without the ability 
to count certifications, licenses, and certificates, it is difficult to accurately assess their role 
in social and economic outcomes; and difficult to evaluate public policies designed to help 
Americans access and benefit from these credentials. 
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These concerns are not new to the federal statistical community. In early 2000, a 
Federal Interagency Committee on Educational Attainment was formed to consider 
improving the measurement of educational attainment. The committee recognized the 
federal interest in developing survey questions to accurately count and classify educational 
certificates and industry-recognized certifications, and enable analysis of the connection 
between certification and employment outcomes. The committee acknowledged that it 
would take research and testing to develop a set of survey questions to address these 
issues, but because of the technical complexity of the topic and a lack of sustained policy 
interest, no further action was taken. 

In February 2009, President Obama asked every American “...to commit to at least 
one year or more of higher education or career training. This can be a community college or 
a four-year school, vocational training, or an apprenticeship. But whatever the training may 
be, every American will need to get more than a high school diploma.” 1 This proposal — and 
similar calls from foundations and other organizations — led to renewed interest in 
certificates and industry-recognized certifications. 

In the fall of 2009 — at the request of the Council of Economic Advisors (CEA), the 
Office of Management and Budget (OMB), and the Under Secretary of Education (OUS) — 
the National Center for Education Statistics (NCES) began a new interagency effort to 
improve federal statistical data on the education, training, and credentials that out-of-school 
youth and adults need for jobs. An Interagency Working Group on Certificates and 
Certifications was founded to oversee this work. In 2012, the group’s scope expanded to 
include measures of participation and enrollment in federal surveys and its name changed 
to the Interagency Working Group on Expanded Measures of Enrollment and Attainment 
(GEMEnA). GEMEnA consists of senior representatives from the Bureau of the Census, the 
Bureau of Labor Statistics (BLS), the National Science Foundation (NSF), as well as CEA, 
NCES, OMB, and OUS. 2 With GEMEnA’s expert guidance and support, NCES has 
embarked upon a multifaceted effort to apply best-practice survey development principles 
towards the goal of developing valid national measures of the participation in and 


1 Address to Joint Session of Congress, February 24, 2009. Full text available at: 

http://www.whitehouse.gov/the press office/Remarks-of-President-Barack-Obama-Address-to-Joint-Session-of- 
Conqress/ . 

2 More information on GEMEnA is provided in appendix A. 
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credentialing of education and training for work, and to build government-wide consensus 
for the adoption of these measures. 

GEMEnA’s portfolio includes four main strands of work, described in more detail in 
chapter 4: 

1 . Develop and deploy a core set of survey items related to the prevalence and key 
characteristics of industry-recognized certifications and occupational licenses. 

2. Develop and deploy a core set of survey items related to the prevalence and key 
characteristics of subbaccalaureate educational certificates. 

3. Consider new and revised measures of participation in education and training 
designed to prepare out-of-school youth and adults for work. 

4. Support NCES in the development of a new household study on education, training, 
and credentials for work. 

The work described in this pilot study is in support of strands 1 and 2 — to develop 
and deploy a core set of survey items related to the prevalence and key characteristics of 
industry-recognized certifications, occupational licenses, and subbaccalaureate educational 
certification. The results of this pilot study also inform strand 4 — to develop a new 
household study on education, training, and credentials for work. 

1.2 Definitions 

GEMEnA began its work with a review of the research literature on education and 
training credentials and the development of a list of existing survey items on certifications, 
licenses, and educational certificates. Based on this background research and discussions 
with experts, certifications and licenses were found to be distinct from educational 
certificates in that they are a job or occupational requirement meant to demonstrate that the 
holder has the authority or skill qualifications necessary to perform specific job duties; 
educational certificates are awarded to show completion of a program of study — which may 
help in the performance of a specific job, but is not necessarily required or considered proof 
of qualification. This review led to the following working definitions of these credentials for 
the purposes of pilot testing: 
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Certification: A credential awarded by a certification body based on an individual 
demonstrating, through an examination process, that he or she has acquired the designated 
knowledge, skills, and abilities to perform a specific job. The examination can be written, 
oral, or performance-based. Certification is a time-limited credential that is renewed through 
a recertification process. 

License: A credential awarded by a licensing agency based on predetermined 
criteria. The criteria may include some combination of degree attainment, certifications, 
certificates, assessment, apprenticeship programs, or work experience. Licenses are time 
limited and must be renewed periodically. 

Educational certificate: A credential awarded by a training provider or educational 
institution based on completion of all requirements for a program of study, including 
coursework and tests or other performance evaluations. Certificates, as an academic 
award, are not time limited and do not need to be renewed. Most educational certificates 
are awarded at the subbaccalaureate level, but a small number are awarded after the 
completion of a postsecondary degree. Certificates of attendance or participation are not in 
the definitional scope for educational certificates. 

As the definitions illustrate, although these credentials have specific characteristics 
and definitions, they are not mutually exclusive. Individuals may enroll in a certificate 
program to gain the knowledge needed to attain a certification which may be required prior 
to applying for a license. The interrelated nature of these credentials was important to 
consider when developing and evaluating potential survey measures. 

GEMEnA’s next step was to engage in a process of survey item development 
designed to yield a short series of measures to enumerate adults with certifications, 
licenses, or certificates that could be used in existing federal household data collections. 
This process led to the development of the ATES Pilot Study questionnaire. 

1.3 The ATES Pilot Study 

The 15-minute ATES Pilot Study questionnaire was developed based on the results 
of focus groups, in-depth cognitive interviews, and GEMEnA input. Four major findings from 
the focus groups and cognitive interviews influenced key decisions during the questionnaire 
design. First, the focus groups showed that the terms “certification” and “license” are not 


5 



always distinguishable to respondents. As noted above, certification is often the process by 
which someone gains the prerequisite qualifications needed to be awarded a license, and 
several focus group participants did not make a distinction between the process and the 
outcome. Consequently, the decision was made to ask about certification and licensure 
jointly. 


Second, focus groups confirmed that certification and license holders do not view 
their certifications or licenses as educational credentials. When asked where they would 
place a certification on their resume, respondents indicated it would be placed under a 
header such as Professional Qualifications or Professional Certifications. Consequently, 
certifications/licenses were not asked about within the questionnaire section on educational 
attainment. 

Third, the cognitive interviews showed that respondents also had trouble 
distinguishing between the terms “certification” and “certificate.” These two terms sound 
similar and, in fact, are shown as synonyms in the Merriam-Webster dictionary (2012). The 
cognitive interviews revealed that some respondents who have a certification will answer 
that they have a “certificate” if the questionnaire asked about certificates before 
certifications. Therefore, the questionnaire was designed to specifically ask about 
certifications and licenses first and certificates second. 

Fourth, respondents’ interpretation of the term “certificate” was varied and broad 
when given little or no definitional context. Accordingly, a definition of certificate, as well as 
examples of certificate types and appropriate certificate providers were included in the lead 
item asking whether or not a respondent has a certificate. 

The focus groups and cognitive interviews also helped to identify key factors 
associated with certifications, licenses, and certificates, and these factors formed the 
operational constructs for the remainder of the questionnaire items. They included the level 
of effort to obtain the credential, examination and performance assessment requirements, 
type of provider or awarding body, industry recognition, occupational specificity, 
requirement for employment, suspension or revocation potential, continuing education and 
recertification requirements, and perceived labor-market value. As appropriate to the 
credential, these factors were collected for respondents’ most recent certification/license 
and certificate. 
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Finally, the questionnaire collected information about respondents’ demographic 
characteristics, such as age, gender, employment status, race/ethnicity, and income. These 
questions were derived from the NCES National Household Education Survey (NHES) and 
the Current Population Survey (CPS). 

The ATES Pilot Study was administered from September 2010 through January 
201 1 . It is important to note that the pilot study was not conducted for the purpose of 
collecting data to make population estimates (although survey weights were developed to 
examine the reasonableness of population estimates generated from the survey measures). 
The primary purpose of the pilot study was to examine the empirical properties of potential 
survey measures. 

The ATES Pilot Study was conducted by mail and telephone using an address- 
based sample study covering the 50 states and the District of Columbia. A two-phase study 
was chosen to improve coverage of all households in the United States through the use of 
the address-based sample at the first phase, while allowing the topical study to be 
conducted by telephone at the second phase to match the mode of administration used 
by the CPS. The study was sponsored by NCES and designed and implemented by the 
American Institutes for Research (AIR) and Westat. The ATES Pilot Study data file contains 
a nationally representative sample of 3,730 adults (aged 18 or older), of whom 
1,230 reported certifications/licenses and 530 reported certificates (220 reported both). 3 In 
addition to the nationally representative sample, the ATES Pilot Study data file contains a 
convenience, or “seeded,” sample of 340 adults, some of whom were voluntarily reported by 
three community colleges to have been awarded a certificate, and others reported by 
five credentialing bodies to have received a certification or license. 4 The seeded sample 
contains state license holders in public health; electrical, pharmaceutical, plumbing, real 
estate, and energy-related fields; certifications/licenses in lab technology, radiation, and 
other health-related fields; and community college certificates covering a variety of 
programs. The seeded sample is not a representative sample and was used primarily to 
assess underreporting of credentials. Although care was taken to find current and reliable 
administrative records for the seeded sample by requesting records 


3 The pilot study data are not available for secondary analysis due to their preliminary nature, their methodological 
rather than reporting purpose, and disclosure concerns. 

4 It is not possible to identify the specific credentialing bodies for disclosure reasons. 
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directly from awarding bodies, these records have unknown measurement error, and 
therefore, at least some portion of the reporting error discussed in the findings of this report 
may be attributed to errors in the seeded sample frame. 

Finally, in a subsample of approximately 1,000 households with two or more adults, 
the responding adult was also asked to report on certifications, licenses, and certificates 
held by another adult in the household. This was done in order to examine measurement 
error for responses reported by proxy because federal surveys that may be candidates for 
the inclusion of these new items may use proxy respondents when they are unable to 
interview the sampled adult in the household. Appendix B contains detailed information 
about the study methodology. 

The first stage of data collection employed a mailed screener that was used to 
enumerate all adults in the household for sampling purposes. The response rate for the 
mailed screener was 52 percent. The second stage of data collection involved a telephone 
interview to conduct the extended survey. The response rate for the second stage was 44 
percent (unweighted) and 42 percent (weighted). It should be noted that limited refusal 
conversion was undertaken at the telephone stage in order to maintain timeliness of the 
data collection. Timeliness and sample yield were a higher priority for the pilot study than 
were response rates. 

The data file used for the analyses in this report underwent limited data editing. 
Imputation was performed on variables needed for weighting and on the overall 
certification/license and certificate items. Weights and variables to calculate standard errors 
were developed and used to examine the reasonableness of population estimates 
generated from the survey items. 

1.4 Objectives of This Report 

This report summarizes the analyses performed on the ATES Pilot Study data file. 
These analyses were designed to answer the primary research question: 

What are the key survey measures that must be asked to generate valid 

population estimates of the prevalence of certifications and certificates in the 

U.S. adult population? 
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In order to answer this question, the report discusses the following questions: 

1 . Do the population estimates generated from the ATES Pilot Study measures seem 
reasonable? 

2. To what extent do respondents misreport certifications, licenses, and certificates, 
and what are the characteristics of respondents and credentials associated with 
misreports? To what extent can proxy respondents provide valid answers to 
questions about certifications, licenses, and certificates? 

1.5 Structure of This Report 

The remainder of this report consists of three chapters, plus seven appendices, as 
described below. 

Chapter 2 presents ATES Pilot Study estimates of the prevalence of 
certifications/licenses and certificates among U.S. adults, overall and by respondent 
characteristics (age, sex, race/ethnicity, educational attainment, industry, labor force 
participation, and immigration status). The chapter then examines the reasonableness of 
the ATES estimates by comparing them, where possible, to estimates from existing data 
sources. The chapter concludes by examining the extent to which the properties of 
certifications/licenses and certificates enumerated in this survey were consistent with key 
factors found to be associated with each type of credential in background research and 
cognitive interviews. 

Chapter 3 presents an analysis of misreporting. This analysis first compares the 
seeded sample records (the administrative list of certification, license, and certificate 
holders from credentialing bodies) against survey reports to examine the extent to which 
respondents underreport (fail to report) their certifications/licenses or certificates. The 
analysis considers levels of misreporting by respondent and credential characteristics. Next, 
using data from the seeded and national samples, the chapter analyzes credential 
commonalities, which can be considered the empirical “defining characteristics” of 
credentials. This analysis examines the information provided by seeded sample 
respondents about their certifications/licenses and certificates to identify characteristics that 
were shared by a high proportion of certifications/licenses and certificates, respectively. The 
analysis then determines whether the commonalities identified in the seeded sample can be 
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used to identify possible instances of overreporting of credentials in the national sample 
(i.e., to determine whether someone who reports a certification/license or certificate 
describes the same key characteristics of that credential as do known credential holders). 
Chapter 3 ends with an examination of the feasibility of proxy reports by analyzing the 
frequency of “don’t know” and other responses to the certification/license and certificate 
items. 


The report concludes with chapter 4, containing GEMEnA’s recommendations for a 
parsimonious set of survey items to measure certifications/licenses and a description of 
work planned under strands 2 through 4. 

The report also includes the following supporting appendices: 

• A. Details on the Interagency Working Group on Expanded Measures of Enrollment 
and Attainment (GEMEnA) 

• B. ATES Pilot Study Design and Methodology 

• C. Supplemental Tables 

• D. Standard Error Tables 

• E. ATES Pilot Study Annotated Extended Interview Questionnaire 

• F. ATES Focus Group Report 

• G. ATES Cognitive Interview Report 
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2. Prevalence of Certifications/Licenses and 
Certificates in the Population 

This chapter presents estimates of the overall rates of certifications/licenses and 
certificates among U.S. adults in 2010-11 and examines how those rates varied across 
subgroups of adults. 5 In addition, the estimates derived from the ATES Pilot Study are 
compared to estimates from other nationally representative data collections. A final section 
examines responses to items found to be associated with each type of credential during 
background research and cognitive testing, such as certification/license holders’ reports that 
obtaining their credential required passing a test or examination. Taken together, these 
analyses provide an indication of the validity of responses obtained from the ATES Pilot 
Study items. The estimates reported in this chapter are provided for the purpose of 
evaluating the questionnaire items, and should not be used to generate or cite population 
estimates. The sample design and data collection procedures were not intended for that 
purpose. 

2.1 Overall Rates of Certifications/Licenses and Certificates 

Responses from the ATES sample were weighted to represent all U.S. adults ages 
18 and older. 6 As shown in table 2-1, an estimated 38 percent of adults, or approximately 
82 million adults, reported having at least one of the credentials of interest. A certification or 
license was more prevalent than a certificate; 30 percent of adults reported having a 
certification or license, compared to 14 percent who reported having a certificate. 7 

Credential holders were also asked to report how many of each type of credential 
they held. Most certification/license or certificate holders reported having one or two 
credentials of the same type (table 2-2) and few reported having three or more 
certifications/licenses or certificates. 


5 Note that the estimates provided in chapter 2 exclude proxy respondents. 

6 See appendix B for details on methods for calculating sampling errors and methods for calculating and applying 
weights, appendix C for tables of unweighted sample estimates, and appendix D for standard error tables. 

7 Although the primary interest of the ATES Pilot Study was on subbaccalaureate certificates, the survey instrument 
did not distinguish the level of the certificate, so respondents may have reported postbaccalaureate certificates. 
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Table 2-1. Number and percentage distribution of respondents reporting a 

certification/license or certificate, by type of credential reported: 2010-11 


95 percent confidence interval 


Type of credential reported 

Number 

Percent 

Lower bound 

Upper bound 

Either credential(s) reported 

82,348,000 

37.5 

35.8 

39.3 

Certifications/licenses 

65,324,000 

29.8 

28.1 

31.5 

Certificates 

29,901,000 

13.6 

12.3 

14.9 

No credentials were reported 

137,076,000 

62.5 

60.7 

64.3 


NOTE: Figures represent weighted estimates. Respondents were able to select multiple credentials; therefore, details may not 
sum to totals. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 


Table 2-2. Number and percentage distribution of reported credentials, by type and 
number of credentials: 201 0-1 1 


Type and number of credentials 
reported 

Number 

Percent 

95 percent confidence interval 
Lower bound Upper bound 

Total 

219,424,000 

100.0 

t 

t 

Certifications/licenses 

None 

153,180,000 

69.8 

68.1 

71.5 

One or more 

65,065,000 

29.7 

28.0 

31.3 

1 

41,188,000 

18.8 

17.3 

20.3 

2 

14,309,000 

6.5 

5.7 

7.4 

3 

5,515,000 

2.5 

1.9 

3.1 

4 

1,628,000 

0.7 

0.5 

1.0 

5 or more 

1,758,000 

0.8 

0.5 

1.1 

Number not specified 

668,000 

# 

T 

T 

Don't know/refused 

1,179,000 

0.5 

0.2 

0.8 

Certificates 

None 

188,480,000 

85.9 

84.6 

87.2 

One or more 

29,852,000 

13.6 

12.3 

14.9 

1 

21,047,000 

9.6 

8.4 

10.8 

2 

3,768,000 

1.7 

1.2 

2.2 

3 

2,130,000 

1.0 

0.6 

1.3 

4 

470,000 

# 

T 

T 

5 or more 

1,777,000 

0.8 

0.4 

1.2 

Number not specified 

661,000 

# 

t 

T 

Don't know/refused 

1,091,000 

# 

T 

T 


# Rounds to zero, 
t Not applicable. 

NOTE: Figures represent weighted estimates. Details may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 
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Among all adults who reported holding a certification, license, or certificate, 

64 percent reported holding a certification/license only, 21 percent reported holding a 
certificate only, and 16 percent reported holding both types of credentials (table 2-3). In 
addition, an analysis of the overlap between certifications/licenses and certificates found 
that 31 percent of certificates were obtained as part of the coursework or training taken for a 
certification or license (not shown in tables). However, because certifications/licenses and 
certificates are distinct types of credentials, they were treated separately in the pilot study 
and the report. 

Similarly, although the cognitive interviews indicated that some respondents used 
“certification” and “license” interchangeably, an analysis was conducted to determine how 
they would respond if prompted for more information about their specific credential type. 

The analysis found that when respondents who reported a certification or license were 
asked if their credential was a certification, a license, or both, 45 percent (unweighted) 
responded “certification only,” 29 percent responded “license only,” and 22 percent 
responded “both” (4 percent of responses were missing; not shown in tables). 

2.2 Rates of Certifications/Licenses and Certificates by Subgroup 

Table 2-3 presents the percentages of adults holding one or both types of 
credentials, broken out by subgroup. The percentage of adults who reported holding any 
credential (a certification/license, and/or a certificate) did not vary measurably by sex, 
race/ethnicity, or immigration status; however, there were significant differences by age 
group, educational attainment level, and labor force status. A smaller percentage of the 
youngest and oldest adults held a credential compared to adults in the middle age ranges; 
for example, adults ages 45 to 54 reported higher rates of any credential than adults ages 
18 to 24 and adults ages 75 or older (47 percent vs. 13 and 23 percent, respectively). 
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Table 2-3. Number and percentage distribution of respondents reporting any credentials, percentage of respondents 
reporting any certification/license or certificate, and among those reporting any credentials, percentage 
distribution reporting a certification/license or certificate only, or both credentials, by respondent 
characteristics: 2010-11 




Percentage distribution of 
respondents who reported 

Percentage of 
respondents who reported 

Among those who reported any credentials, 
percentage distribution of respondents who 
reported 



No 

Any 

Any 

certification/ 

Any 

Certification/ 

Certificate 

Both a certification/ 
license and 

Respondent characteristics 

Number 

credentials 

credentials 

license certificate 

license only 

only 

certificate 

Total 

219,424,000 

62.5 

37.5 

29.8 

13.6 

63.7 

20.7 

15.6 

Sex 

Male 

105,957,000 

62.9 

37.1 

30.3 

12.7 

65.8 

18.4 

15.8 

Female 

113,467,000 

62.1 

37.9 

29.3 

14.5 

61.8 

22.8 

15.5 

Race/ethnicity 1 

White 

151,856,000 

61.5 

38.5 

31.2 

13.2 

65.6 

18.9 

15.4 

Black 

23,306,000 

62.7 

37.3 

26.8 

16.9 

54.8 

28.2 

17.0 

Hispanic 

29,027,000 

62.7 

33.3 

26.6 

12.7 

62.0 

20.1 

17.9 

Asian 

5,844,000 

68.3 

31.7 

26.7 

9.7 

69.4 

15.6 

15.1 

Other 

2,974,000 

62.1 

37.9 

26.7 

13.3 

64.9 

29.5 

5.6 

Two or more races 

6,416,000 

61.2 

38.8 

24.4 

19.2 

50.7 

37.1 

12.2 

Age 

18 to 24 years old 

28,799,000 

87.3 

12.7 

11.6 

2.7 

78.5 

8.9 

12.5 

25 to 34 years old 

37,102,000 

58.8 

41.2 

34.1 

12.5 

69.7 

17.4 

12.9 

35 to 44 years old 

38,192,000 

55.5 

44.5 

37.2 

13.9 

68.8 

14.2 

16.9 

45 to 54 years old 

40,629,000 

53.1 

46.9 

36.4 

19.6 

58.2 

22.2 

19.5 

55 to 64 years old 

34,498,000 

57.0 

43.0 

31.9 

17.1 

60.3 

25.8 

13.9 

65 to 74 years old 

22,186,000 

62.1 

37.9 

28.0 

15.6 

58.9 

26.3 

14.8 

75 years old and over 

18,018,000 

77.2 

22.8 

15.3 

10.5 

54.1 

32.9 

13.0 


See notes at end of table 
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Table 2-3. Number and percentage distribution of respondents reporting any credentials, percentage of respondents 
reporting any certification/license or certificate, and among those reporting any credentials, percentage 
distribution reporting a certification/license or certificate only, or both credentials, by respondent 
characteristics: 2010-11 — Continued 




Percentage distribution of 
respondents who reported 

Percentage of 
respondents who reported 

Among those who reported any credentials, 
percentage distribution of respondents who reported 

Respondent characteristics 

Number 

No 

credentials 

Any 

credentials 

Any 

certification/ Any 

license certificate 

Certification/ 
license only 

Certificate 

only 

Both a certification/ 
license and 
certificate 

Highest reported level of 
education 









Less than high school 

33,485,000 

79.0 

21.0 

14.5 

9.4 

55.1 

30.7 

14.3 

Grades 1-11 

23,931,000 

81.8 

18.2 

12.9 

8.8 

51.9 

29.4 

18.8 

12th grade (no diploma) 

9,554,000 

72.1 

27.9 

18.8 

11.1 

60.3 

32.8 

6.9 

High school (or equivalent) 

67,299,000 

72.9 

27.1 

20.3 

11.7 

56.8 

25.1 

18.0 

Regular high school 
diploma 

59,781,000 

72.6 

27.4 

20.6 

11.9 

56.8 

24.9 

18.3 

GED or alternative 
credential 

7,518,000 

75.3 

24.7 

18.1 

10.5 

57.6 

26.9 

15.5 

Some college or associate’s 
degree 

58,385,000 

54.4 

45.6 

32.1 

22.5 

50.6 

29.7 

19.8 

Some college credit, but 
less than 1 year 

11,267,000 

59.5 

40.5 

23.3 

25.4 

37.1 

42.3 

20.5 

1 or more years of 
college credit, no 
degree 

35,054,000 

55.5 

44.5 

31.8 

21.0 

52.9 

28.6 

18.6 

Associate’s degree 

12,064,000 

46.5 

53.5 

41.0 

24.3 

54.6 

23.4 

22.1 

Bachelor’s degree 

36,225,000 

55.1 

44.9 

40.7 

9.0 

79.9 

9.4 

10.7 

Graduate or professional degree 

24,030,000 

41.0 

59.0 

55.4 

10.2 

82.7 

6.1 

11.2 

Master's degree 

17,881,000 

43.4 

56.6 

52.1 

11.2 

80.2 

7.9 

11.9 

Professional degree 
beyond a bachelor's 
degree 

2,690,000 

18.1 

81.9 

80.2 

11.7 

85.7 

2.1 

12.2 

Doctorate degree 

3,458,000 

46.3 

53.7 

53.3 

3.9 

92.8 

0.8 

6.4 


See notes at end of table. 
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Table 2-3. Number and percentage distribution of respondents reporting any credentials, percentage of adults 

reporting any certification/license or certificate, and among those reporting any credentials, percentage 
distribution reporting a certification/license or certificate only, or both credentials, by respondent 
characteristics: 2010-11 — Continued 




Percentage distribution of 
respondents who reported 

Percentage of 
respondents who 
reported 

Among those who reported any credentials, 
percentage distribution of respondents who 
reported 

No 

Any 

Any 

certification/ 

Any 

Certification/ 

Certificate 

Both a certification/ 
license and 

Respondent characteristics 

Number 

credentials 

credentials 

license certificate 

license only 

only 

certificate 

Labor force status 









Employed 

123,556,000 

55.2 

44.8 

37.8 

13.8 

69.3 

15.4 

15.3 

Unemployed (seeking 









employment) 

21,438,000 

73.3 

26.7 

20.4 

12.0 

55.1 

23.8 

21.2 

Not in labor force 

71,369,000 

72.1 

27.9 

18.0 

13.6 

51.1 

35.4 

13.5 

Labor force status unknown 

3,060,000 

52.8 

47.2 

44.4 

20.0 

57.6 

6.1 

36.3 

Immigration status 









Born in U.S. 

196,861,000 

62.6 

37.4 

29.5 

13.6 

63.6 

21.1 

15.3 

Born outside of U.S. 

22,563,000 

61.6 

38.4 

31.9 

13.6 

64.6 

16.9 

18.6 


1 Black includes African American. Hispanic includes Latino. Other includes Pacific Islanders and American Indians (including Alaska Natives), and two or more races includes 
non-Hispanics reporting more than one race. Race categories exclude persons of Hispanic ethnicity. 

NOTE: Figures represent weighted estimates. Credentials include certifications and certificates. Imputed data were used for all respondent characteristics except labor force 
status, for which imputed data were not available. Details may not sum to totals because of rounding. Because a certificate represents an educational attainment level, the findings 
by educational attainment are discussed separately for certifications/licenses and for certificates. In general, certification/licensing rates increased with educational attainment. 
Approximately 13 percent of adults with the lowest level of education (grades 1-11) held a certification/license, and 21 percent of adults with a regular high school diploma held 
one of these credentials. These estimates were lower than those for adults with higher levels of attainment; for example, 32 percent of adults with one or more years of college 
credit (but no degree) reported a certification/license, as did 80 percent of adults with a post-bachelor’s professional degree (e.g., an M.D. or J.D.). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. 
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Looking at certificates, 9 percent of adults with a grades 1-1 1 education held a 
certificate, and 12 percent of adults with a regular high school diploma held one of these 
credentials. 8 A higher percentage of adults with some college (but no degree) or an 
associate’s degree reported holding a certificate (23 percent) than any other group. This is 
consistent with certificates being predominantly a subbaccalaureate credential. Unlike 
certifications/licenses, however, adults with bachelor’s and master’s degrees did not have 
higher certificate rates than those with a high school degree or lower; 9 percent of bachelor 
degree holders and 10 percent of advanced degree holders reported holding a certificate. 

Finally, employed adults reported higher rates of credentials than did unemployed 
adults and those not in the labor force (45 percent compared to 27 and 28 percent, 
respectively, for any credential). Looking separately at certifications/licenses and 
certificates, however, the differences remained significant only among certification/ 
license holders. 


8 This finding is somewhat puzzling because we would anticipate that respondents who reported their highest level of 
educational attainment to be a high school diploma or lower would not have also reported that they had a 
subbaccalaureate educational certificate from a postsecondary institution. There are at least two possible 
explanations for this finding: (1) The main educational attainment item did not have a “certificate” response option; 
although respondents could have selected a “some college” option, some certificate holders may not have perceived 
that option as relevant to them; and (2) The main certificate item could be capturing noneducational certificates that 
people obtained through other types of organizations, such as the workplace. These possibilities will be investigated 
in future research studies on certificates. 
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2.3 Comparison of Estimates From the ATES Pilot Study to Previous 
Data Collections 


To determine whether the ATES Pilot Study estimates were consistent with past 
research, the ATES estimates were compared to estimates derived from the Princeton Data 
Improvement Initiative (PDII) for certifications/licenses and to the Survey of Income and 
Program Participation (SlPP)for certificates. 

The PDII survey was conducted in 2008 by Westat on behalf of Princeton University 
and covered the topics of worker perceptions of the “offshoreability” of their jobs, 
occupational licensing, and lifetime work experience. 9 Participants were adults in the labor 
force, currently employed or unemployed and looking for work. After restricting the ATES 
and PDII samples to adults who were currently employed, the percentage of adults 
reporting a certification/license was 38 percent based on the ATES Pilot Study data and 
34 percent based on the PDII data (table 2-4). Although the ATES estimate was measurably 
different from the PDII estimate, the difference is small enough to be considered 
reasonable, given dissimilarities in the data sets in terms of timing, item wording, and item 
order. When broken out by educational attainment level, the rate of certifications/licenses 
was not measurably different for adults whose highest level of education was below a 
bachelor’s degree, but was measurably higher in ATES than in the PDII for those at the 
bachelor’s degree level and above (by around 7 to 8 percentage points). 

To examine the ATES estimates on certificates, the ATES data were reanalyzed and 
compared to 2008 data from the U.S. Census Bureau’s SIPP, using adults ages 23-64 who 
reported an income or reported working for earnings in the past year. 10 The overall rate of 
certificates based on ATES was 15 percent, compared to 18 percent for SIPP. Again, the 
difference was statistically significant, but reasonable given the dissimilarities between the 
two surveys (e.g., timing, mode of interview). Significant differences also existed between 
the two surveys for adults with educational attainment levels at or below high school 
level, and at the graduate or professional degree level (differences in the range of 2 to 
6 percentage points). There were no measurable differences in the estimates for adults with 
some college, an associate’s degree, or a bachelor’s degree. 


9 The PDII data are available at http://irs.princeton.edu/Conferences/PDII/PDII RDD Survey. dta . 

10 The SIPP 2008 data are available at http://www.bls.census.gov/sipp ftp.html . 
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2.4 Unweighted Responses to Items Related to Certifications/Licenses 
and Certificates 

Based on ATES developmental work, several factors were expected to covary with 
holding a certification/license or certificate. First, certification/license holders were expected 
to report that their credentials were in some way job relevant; for example, respondents 
might report that their certifications/licenses were required for their current job. 

Respondents were also expected to report taking courses or training to earn the 
certification/license and that they had been required to demonstrate their knowledge or 
skills in order to be certified/licensed. Finally, respondents were expected to report being 
certified or licensed by a state, industry, company, or professional association and that their 
certification/license could be revoked or suspended. Factors expected to be related to 
certificates reflected the fact that this type of credential is obtained through an educational 
program. Therefore, certificate holders were expected to report that their credentials were 
awarded by an educational institution (e.g., a community college). 

The remainder of this chapter summarizes the findings from an analysis of 
unweighted responses to items asked of credential holders on the factors described above. 
Respondents were asked some items, such as the name and year the credential was 
obtained, for all certifications/licenses or certificates held (up to five of each), whereas the 
majority of credential-specific items were only asked for the most recent certificate earned 
and the most recent work-related certification/license held . 11 Because information was 
collected on potentially multiple credentials per respondent, items were analyzed at the 
credential level rather than the respondent level and are reported as the percentage of 
certifications/licenses or certificates held by respondents. In addition, these analyses are 
not weighted because, unlike comparisons of estimates, they are methodological analyses 
of sample characteristics and are not intended to show population characteristics. 


11 If respondents reported any certifications/licenses but did not report any work-related certifications/licenses, they 
were asked a set of follow-up questions about the most recent credential they obtained for personal interest. 
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Table 2-4. Percentage of adults who reported holding certifications/licenses in the Princeton Data Improvement 
Initiative (PDII) and the Adult Training and Education Survey (ATES) Pilot Study and percentage of 
adults who reported holding a certificate in the Survey of Income and Program Participation (SIPP) and 
the Adult Training and Education Survey (ATES) Pilot Study by educational attainment level: 2008, 
2009, and 2010-11 


Certifications/licenses Certificates 

ATES 95 percent ATES 95 percent 

confidence interval confidence interval 





Lower 

Upper 



Lower 

Upper 

Educational attainment level 

PDII 

ATES 

bound 

bound 

SIPP:08i 

ATES 

bound 

bound 

Total 

34.4 

37.8* 

35.5 

40.2 

18.1 

15.1* 

13.2 

17.0 

Highest level of education 









Less than high school diploma 

18.5 

27.2 

17.5 

36.8 

7.3 

11.7* 

5.2 

18.3 

High school diploma 

25.7 

27.8 

23.6 

32.1 

22.2 

15.8* 

12.0 

19.6 

Some college 

33.9 

38.1 

33.0 

43.2 

24.8 

23.2 

17.3 

19.1 

Associate’s degree 

— 

— 

— 

— 

34.3 

24.3 

15.4 

33.2 

Bachelor’s degree 

35.1 

42.7* 

37.7 

47.6 

10.3 

9.4 

7.0 

11.9 

Graduate or professional degree 

50.3 

56.8* 

50.5 

63.2 

6.4 

8.8* 

5.1 

12.6 


— Not available. 

* p < 0.05. 

1 Survey of Income and Program Participation, 2008. 

NOTE: Figures represent weighted estimates. Data for certifications/licenses limited to those currently employed or looking for employment. Data for certificates limited to those 
who were between 23 and 64 years of age and reported an income or earnings in the previous year. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11; U.S. Department of 
Commerce, Census Bureau, Survey of Income and Program Participation (SIPP), 2008; Kleiner, M.M., and Krueger, A.B., Analyzing the Extent and Influence of Occupational 
Licensing on the Labor Market, Princeton Data Improvement Initiative (PDII), 2009. 
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2.4.1 Findings related to certifications/licenses held 

Certifications/licenses are occupation and/or industry specific and were therefore 
expected to be related to — and in some cases required for — respondents’ jobs (e.g. medical 
licenses). The ATES pilot test results supported this expectation: the majority of 
certifications/licenses held by respondents were pursued for mainly work-related reasons 
(90 percent), while 8 percent were pursued for mainly personal interest (table 2-5). 

Similarly, for the most recent work-related certification/license reported by currently 
employed respondents, 82 percent were related to their current job, and 60 percent were 
required for their current job (table 2-6). 

Adults can prepare for a certification/license by taking courses and/or training or 
through self-study. Among the ATES Pilot Study sample, 90 percent of the most recent 
work-related certifications/licenses were earned after taking some type of courses and/or 
training (table 2-7). In addition, 91 percent of the most recent work-related certifications/ 
licenses required respondents to pass a test or exam, 66 percent required a demonstration 
of skills while on the job, and 31 percent required the submission of a portfolio of work. 
Overall, 96 percent of these credentials had one or more of these performance evaluation 
requirements. To maintain the credential, 64 percent of certifications/licenses required 
respondents to take continuing education units in order to maintain them, 31 percent 
required periodic testing, and 70 percent had one or both of these requirements. 


Table 2-5. Percentage distribution of certifications/licenses pursued for 
mainly work-related reasons or for personal interest: 2010-11 


Reason for pursuing certification/license 

Percent 

Mainly work-related 

90.2 

Mainly personal interest 

7.9 

Data missing or not collected 

1.9 

Don't know 

1.1 

Refused 

# 

Not collected 

# 


# Rounds to zero. 

NOTE: Figures represent unweighted estimates. Survey question was asked for the first five certifications/licenses 
identified per respondent. Includes self-reported data only; data completed by proxy were excluded from the analysis. 
Details may not sum to total because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education 
Survey (ATES) Pilot Study, 2010-11. 
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Table 2-6. Percentage distribution of certifications/licenses related to 
respondents’ current jobs: 2010-11 


Certification/license relatedness to current job 

Percent 

Related to current job 

81.9 

Required for current job 

59.5 

Not required for current job (include missing) 

22.3 

Not related to current job (include missing) 

18.1 

NOTE: Figures represent unweighted estimates. Survey questions were asked only for the most recent work-related 
certification/license or, if none were identified as work related, for the most recent certification/license. Additionally, this 
survey question was only asked of those respondents who reported being currently employed. Includes self-reported 
data only; data completed by proxy were excluded from the analysis. Details may not sum to total because of rounding. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education 

Survey (ATES) Pilot Study, 2010-11. 


Table 2-7. Percentage of the most recent certifications/licenses requiring 

courses or training, performance evaluations, and ongoing 
maintenance: 2010-11 


Percent of 

Certification/license requirements 

certifications/licenses 

Course or training requirements 


Took courses or training, or both 

90.0 

Did not take courses or training 

9.6 

Performance evaluation requirements 


Demonstration of skills while on-the-job 

65.7 

Passing a test or exam 

91.0 

Submitting a portfolio of work 

31.4 

Any of the above 

Ongoing maintenance requirements 

96.0 

Continuing education classes or units (CEUs) 

63.6 

Periodic tests 

31.0 

Any of the above 

69.6 


NOTE: Figures represent unweighted estimates. Survey questions were asked only for the most recent work-related 
certification/license or, if none were identified as work-related, for the most recent personal interest certification/license. 

Includes self-reported data only; data completed by proxy were excluded from the analysis. Respondents were able to 
select multiple requirements; therefore, estimates are not mutually exclusive. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

Certifications/licenses can be conferred by states, industries, companies, and 
professional associations. Among the most recent work-related certifications/licenses 
obtained by the ATES Pilot Study respondents, most were issued either by a state 
(68 percent) or a professional association (16 percent) (table 2-8). Although 14 percent of 
certifications/licenses were reported to have an “other” provider, analysis of the verbatim 
responses to the “other” category suggest that adding response options for local and federal 
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governments could capture 37 percent of these “other” responses (or about 5 percent of all 
responses) (not shown in tables). 

Finally, respondents indicated that 76 percent of their certifications/licenses could be 
revoked or suspended (e.g., for not meeting continuing education requirements). 

Table 2-8. Percentage of most recent certifications/licenses, by issuing 

organization and whether certification/license can be revoked or 


suspended: 2010-11 

Certification/license characteristic 

Percent of 
certifications/licenses 

Organization issuing certification/license 


State 

67.8 

Industry 

2.6 

Company 

5.1 

Professional association 

15.9 

Other 

13.8 

Don't know 

1.6 

Certification/license can be revoked or suspended 


Certification/license can be revoked or suspended for any reason 

76.3 

Certification/license cannot be revoked or suspended for any reason 

14.8 

Don't know 

8.9 


NOTE: Figures represent unweighted estimates. Survey questions were asked only for the most recent work-related 
certification/license or, if none were identified as work-related, for the most recent certification/license. Includes self- 
reported data only; data completed by proxy were excluded from the analysis. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

2.4.2 Findings related to certificates held 

The purpose of the ATES certificate items was to measure the prevalence of 
educational certificates: that is, certificates earned as part of an educational program of 
study, as opposed to certificates of attendance, achievement, or completion that were not 
part of a program of study. Therefore, it was expected that the organization conferring the 
certificate would be a type of educational institution — a community college, another type of 
university or college, or a trade school. Table 2-9 shows that overall, 57 percent of the 
certificates reported by ATES Pilot Study respondents were awarded by these types of 
institutions, with 21 percent being earned from a community college, 15 percent from 
another type of university or college, and 22 percent from a trade school (figures do not sum 
to the total due to rounding). Fourteen percent of the remaining certificates were conferred 
by other specified types of organizations, and 26 percent were reportedly obtained from a 
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type of organization not listed in the survey. However, an analysis of verbatim responses for 
the “other” category found that 50 percent of those cases could have been categorized as 
an educational institution. Adding these cases to the educational provider categories would 
raise the percentage of certificates awarded by educational institutions from 57 to 73 (not 
shown in tables). 


Table 2-9. Percentage distribution of most recent certificate, by source of 
certificate: 2010-11 


Source of certificate 

Percent of certificates 

A community college 

20.5 

Another university or college 

14.7 

A trade school 

21.5 

Business or company 

5.2 

Professional association 

2.9 

Trade union 

1.6 

Nonprofit organization 

0.8 

A federal, state, or local government 

3.5 

Other 

25.6 

Don't know, refused, or missing 

3.7 


NOTE: Figures represent unweighted estimates. Survey question was asked only for the most recent certificate. 

Includes self-reported data only; data completed by proxy were excluded from the analysis. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

The number of credits or course hours required for an educational certificate was 
also expected to vary to some extent; however, certificates earned through short-term 
training, such as a 1-day course, were not considered educational certificates, according to 
the ATES definition. As a frame of reference, a minimum of approximately 30 credit hours 
would be required to complete a one-year certificate program. 12 

Overall, only 16 percent of ATES certificate-holders reported credit hour information. 
Among these respondents, 15 percent of the most recent certificates earned were reported 
to require fewer than 5 credit hours to complete, and the majority (57 percent) required 
21 or more credit hours to complete (table 2-10). 13 The average number of credit hours 
reported to earn the most recent certificate was 92; when responses were capped at 


12 This number is based on an informal review of college websites and the guidance provided to institutions 
responding to NCES’s Integrated Postsecondary Education Data System (IPEDS) surveys. IPEDS defines a 
postsecondary award, certificate, or diploma of at least 1 but less than 2 academic years as requiring one of the 
following: (1) at least 900, but less than 2,100 contact or clock hours; (2) at least 30, but less than 60 semester or 
trimester credit hours; or (3) at least 45, but less than 90 quarter credit hours. 
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150 credit hours, the average number of credit hours was 32 (not shown in tables). Over 
half of the certificate holders reported that they did not know the time taken to complete 
their certificate. This, combined with the high credit hours reported by some respondents, 
indicate that this information is difficult for respondents to provide. For example, three 
respondents reported that their most recent certificate required 2,000 credit hours. 


Table 2-10. Percentage distribution of most recent certificates, by number of 
credit hours required to complete certificate: 2010-11 


Number of credit hours required to complete certificate 

Percent 

People who reported in credit hours 

100.0 

1 to 4 credit hours 

14.6 

5 to 8 credit hours 

6.1 

9 to 12 credit hours 

6.1 

13 to 20 credit hours 

15.9 

21 or more credit hours 

57.3 

Data missing or not collected 

100.0 

Don't know 

93.8 

Refused 

1.0 

Missing 

5.2 


NOTE: Figures represent unweighted estimates. Survey questions were asked only for the most recent certificate. 

Includes self-reported data only; data completed by proxy were excluded from the analysis. The average number of 
credit hours needed to earn the most recent certificate was 92. However, inspection of the extreme outliers indicated 
that a reporting error for fewer than 3 respondents reporting an excess of 2000 credit hours skewed the average. When 
credit hour responses were capped at 150, the average number of credit hours needed was 32. Detail may not sum to 
totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

2.5 Summary of Findings on Prevalence of Certifications/Licenses 
and Certificates 

This chapter presented the overall rates of certifications/licenses and certificates 
among U.S. adults in 2010-11 and investigated how these rates varied across subgroups of 
adults and data collections. Overall rates of certifications/licenses and certificates based on 
ATES Pilot Study data were found to be reasonable in comparison with other studies of 
similar populations. The ATES Pilot Study found that 38 percent of U.S. adults reported 
holding at least one credential, with certifications/licenses being reported by a higher 
percentage of adults than certificates — 30 percent reported a certification/license, while 


13 Inspection of the extreme outliers for this question indicated that a reporting error for fewer than 3 respondents 
reporting an excess of 2000 credit hours skewed the average. With the outliers removed, the maximum number of 
credit hours reported was 150. 
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14 percent reported a certificate. These estimates did not vary by sex, race/ethnicity, or 
immigration status; however, they did vary by several background characteristics, including 
age group, educational attainment level, and labor force status, in ways that one might 
expect. For example, younger and older adults were less likely to report holding these 
credentials than were mid-aged adults, and employed adults were more likely to report them 
than were adults who were not employed. 

In addition to producing estimates that were consistent with those from other data 
collections, the ATES Pilot Study found relationships between holding a certification/license 
or certificate and other factors as expected, based on ATES developmental work and the 
definitions of the credentials. For example, most certifications/licenses were pursued for 
mainly work-related reasons (90 percent), and most work-related certifications/licenses 
required the respondent to pass a test or exam (91 percent) and take continuing education 
units to maintain them (64 percent). The majority of most recent certifications/licenses could 
be revoked or suspended (76 percent). Similarly, the majority of most recent certificates 
were conferred by an educational institution (57 to 73 percent) and required 21 or more 
credit hours to complete (57 percent for those reporting in credit hours). Instances where 
these expected relationships were not found might indicate misreporting by respondents; 
this issue is examined in the next chapter. 
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3. Evaluation of Measurement Error 

This chapter presents analyses of misreporting in the ATES Pilot Study. The first 
analysis compares the seeded sample records against survey reports, to examine the 
extent to which respondents underreported their certifications/licenses or certificates 
(i.e., “false negatives”). The analysis evaluates levels of underreporting by respondent and 
credential characteristics. The seeded sample is a convenience sample of adults who were 
reported by three community colleges and five credentialing bodies to have received a 
certification, license, or certificate; it is not a representative sample of any credentialed 
group. The seeded sample contains state license holders in alcohol- and drug-related 
fields and electrical, pharmaceutical, plumbing, real estate, and energy-related fields; 
certifications/licenses in lab technology, radiation, and other health-related fields; and 
community college certificates covering a variety of programs. Please refer to appendix B 
for more information on this sample . 14 

The analysis of credential commonalities follows the examination of underreporting, 
and uses data from the seeded and national samples. First, the analysis examines the 
information provided by confirmed seeded sample respondents (i.e., “true positives”) about 
their certifications/licenses and certificates to identify related factors (e.g., test or exam 
requirements) that were shared by a high proportion of certifications/licenses and 
certificates, respectively. Using this information, the next part of the analysis considers 
whether the commonalities identified using the seeded sample can be used to identify 
possible instances of credential overreporting (i.e., “false positives”) in the national sample. 
Note that the seeded sample is a nonrepresentative convenience sample, and therefore all 
results based on the seeded sample should be considered illustrative only. 

Finally, this chapter provides an analysis of the reasonableness of proxy reports by 
comparing respondent self-reports to proxy reports for each certification/license and 
certificate item and examines the use of help text probes for the main certification/license 
and certificate items. 


14 It is not possible to identify the specific credentialing bodies for disclosure reasons. 
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3.1 Methods 


The ATES seeded sample is a convenience sample of 340 respondents known to 
have specific certifications/licenses or certificates (see appendix B for detailed information 
about the seeded sample). For certifications/licenses, credential holder lists from one state 
(for licenses) and four national accrediting bodies (for certifications) were used to develop a 
list sampling frame that included the name and address of the credential holder and the 
type of credential. 15 A number of people known to have earned certificates from three 
community colleges were also sampled. 16 Although the seeded sample is not representative 
of all credential holders, it does allow us to examine the measurement properties of the 
survey items in three important ways. First, this sample allows for an evaluation of 
underreporting rates for selected certifications/licenses and certificates using reverse record 
checks. Second, the sample can be used to determine whether the self-reported credentials 
match the credentials provided by the credentialing body. Lastly, the characteristics of true 
credentials (those reported by both the respondents and the credentialing bodies) can then 
provide a rule-of-thumb for identifying credential overreports. Credentials reported in the 
nationally representative sample that do not reflect the expected characteristics of 
credentials, as identified using the seeded sample, may not be true credentials. However, 
the validity of this approach is limited by the extent to which the seeded sample may or may 
not reflect the characteristics of all credentials. 

3.2 Analysis of Underreporting 

Reverse record checks, like the one employed in the ATES Pilot Study, are designed 
to measure underreporting of known characteristics. In doing so, seeded samples can be 
used to identify characteristics of respondents and/or credentials that could help explain 
underreporting. This analysis examined differential underreporting rates by educational 
attainment, sex, race, age, and employment industry. Some portion of underreports could 
also be due to errors in the administrative records or data collection, such as reaching the 
wrong individual. 


15 Certification/license types included in the seeded sample were alcohol and drug counselor or aide; health 
technician, specialist, or sonographer; pharmacy technician; electrical aide; plumber; HVAC technician; and fuel 
technician. 

16 Certificate types included in the seeded sample were, among others, accounting; business management; cabinet 
and furniture technology; dental assisting; diesel mechanics technology; drafting technology; graphic communications; 
photography; and real estate. 
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One possible explanation for underreporting is that respondents may misidentify a 
certification/license as a certificate (or vice versa). Differential underreporting rates can be 
used to evaluate this by determining whether a certificate was also reported by the 
respondent (and whether that certificate was from someplace other than a college or other 
educational program, as this would be an indication that the credential was more likely a 
certification/license than a certificate). Similarly, the evaluation of seeded certificates looked 
for differential underreporting rates by whether a certification/license was also reported by 
the respondent and whether those certifications/licenses were part of a degree or other 
educational program. Finally, the evaluation of seeded certifications/licenses assessed 
differential underreporting rates by credential type (certification or license). These additional 
analyses can provide evidence of credential misidentification at an aggregate level. 

A record was classified (at the person level) as a certification/license underreport if a 
respondent who was sampled from the seeded certification/license frame answered “no,” 
“don’t know,” or “refused” to the main certification/license survey question (question CN1: 
Do you have a professional certification or a state or industry license? See appendix E for 
the survey instrument). Similarly, a record was classified (at the person level) as a 
certificate underreport if a respondent who was sampled from the certificate frame 
answered “no,” “don’t know,” or “refused” to the main certificate survey question (question 
CT1: Some people decide to enroll at a college, university, community college, or trade 
school to earn a certificate rather than a degree. Have you ever earned this type of 
certificate?). Proxy responses were not included in this analysis to avoid confounding the 
possibility of limited proxy knowledge with actual underreports. 17 

The overall underreporting rate for seeded certifications/licenses was 15 percent, 
and the overall underreporting rate for seeded certificates was 50 percent. 18 The results 


17 In addition, respondents selected from one of the community colleges in the certificate sample frame were removed 
after finding that they had an unusually high certificate underreporting rate (85 percent); further investigation revealed 
that there was a problem with the sample frame itself. The school had mistakenly encoded students who achieved a 
transfer sequence of 18 to 60 units as “certificates.” These students represented about 78 percent of the records in 
the sampling frame and 86 percent of the survey respondents for this school. The sampled respondents with 
problematic records from this school had an underreporting rate of 93 percent, while the sampled respondents with 
valid records had an underreporting rate of 57 percent. However, while the rate for the latter group is more in line with 
the other certificate samples, the sample size was too small to be included in the analyses (n < 10). 

18 Based on the inaccuracies in the certificate seeded sampling frame described above, it is possible that similar 
(though less extensive) issues existed in data supplied by the other community colleges and that the high 
underreporting rate found for certificates is partially due to characteristics of the frame. This possibility will be 
investigated by NCES in upcoming work on certificate item development and testing. 
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from this analysis are presented in tables 3-1 and 3-2 for certifications/licenses and 
certificates, respectively. The tables report the response distributions for each seeded 
sample and the differential underreporting rates by selected respondent characteristics. 

To help describe the strength of the association between each of the selected respondent 
characteristics and credential underreporting, the tables also provide estimates of the 
unadjusted odds ratios. The unadjusted odds ratios presented in the tables estimate the 
odds that a credential will not be reported given that respondent’s characteristics. To allow 
for comparisons and significance testing across subgroups, the 95 percent confidence 
interval of the odds ratio is also provided in the tables. 

Few of the differences in underreporting rates by subgroup were statistically 
significant, as evidenced by the overlapping confidence intervals for the odds ratios. 
However, additional analyses suggest that labor force status may be driving the 
underreporting results. A multivariate logistic regression model was used to estimate the 
probability of underreporting (a binary outcome variable where 1 indicates an underreport 
and a 0 indicates no underreport), as a function of three predictor variables: work status in 
the last 12 months (1=Has worked; 0=Has not worked), age (1=18-64 years of age; 2=65 or 
more years of age), and sex (1=Male; 2=Female). Logistic regression works by modeling 
the log-odds of a particular binary response as a linear combination of regression 
coefficients, or predictor variables. This analysis modeled the probability that the binary 
response variable (UNDERREPORT) equaled 1, which indicated an underreport. To aid 
interpretability, the coefficients produced through the multivariate logistic regression model 
were converted to adjusted odds ratios, which show the increased odds of underreporting 
attributable to labor force status while all other factors controlled for in the model are held 
constant. Regardless of sex or age group, currently employed respondents were found to 
be the most accurate at reporting on these types of credentials (not shown in tables). 19 For 
example, whereas the underreporting rate for certifications/licenses among males under the 
age of 65 who have worked in the last 12 months was 8 percent, the corresponding rate for 
males under age 65 who have not worked in the past 12 months was 44 percent (not shown 
in tables). 


19 When controlling for age and sex, respondents who have not worked in the past 12 months were 5.1 times more 
likely to omit their certification or license than respondents who have worked in the past 12 months (p = 0.0018). The 
odds ratios associated with age and sex were not significantly different from zero at the p < 0.05 level. A possible 
explanation for this finding is salience of the credential to the employed respondents; most employed respondents 
reported that their credentials were related to their current jobs, and therefore these credentials may be more likely to 
be remembered than credentials that are not currently being used. 
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Table 3-1. Percentage distribution of respondents from the certification/license seeded sample frame and 

percentage and odds ratios of seeded sample respondents who did not report the credential listed 
in the frame (underreporting), by selected respondent characteristics: 2010-11 


Selected respondent characteristics 

Percent of respondents from 
the certification/license 
seeded sample frame 

Percent 

of respondents who did not 
report the credential listed in 
the frame 
(underreporting rate) 

Odds ratio of 
underreporting 

95 percent 
confidence interval 
for odds ratio 

Lower Upper 

bound bound 

Total 

100.0 

15.0 

(i) 

(i) 

(i) 

CN4. Type of certification 

Certification 

38.5 

17.1 

1.3 

0.6 

2.8 

License 

61.5 

13.7 

(i) 

(i) 

(i) 

ED3. Highest reported level of education 

High school or less than high school 

30.8 

16.9 

(i) 

(i) 

(i) 

Some college or associate’s degree 

39.8 

13.1 

0.7 

0.3 

1.8 

Bachelor's, graduate, or professional degree 

29.4 

16.1 

0.9 

0.4 

2.4 

ALl.Age 

18 to 34 years old 

19.8 

19.0 

1.8 

0.7 

4.5 

35 to 64 years old 

68.9 

11.6 

(i) 

(i) 

(i) 

65 years old and over 

11.3 

29.2 

3.1 

1.1 

8.6 

AL3/AL4. Race/ethnicity 

White 2 

92.4 

14.4 

(i) 

(i) 

(i) 

All other races/ethnicities 

7.6 

18.8 

1.4 

0.4 

5.1 

AL6. Sex 

Male 

64.3 

11.7 

(i) 

(i) 

(i) 

Female 

35.7 

21.1 

2.0 

0.9 

4.3 


See notes at end of table. 
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Table 3-1. Percentage distribution of respondents from the certification/license seeded sample frame and 

percentage and odds ratios of seeded sample respondents who did not report the credential listed 
in the frame (underreporting), by selected respondent characteristics: 2010-11 — Continued 


Selected respondent characteristics 

Percent of respondents from 
the certification/license 
seeded sample frame 

Percent of respondents who 
did not report the credential 
listed in the frame 
(underreporting rate) 

Odds ratio of 
underreporting 

95 percent 
confidence interval 
for odds ratio 

Lower Upper 

bound bound 

IN6/IN7. Employment Industry 

Private company 

48.8 

15.4 

(i) 

(i) 

(i) 

Government, nonprofit, or self-employed 

33.8 

6.9 

0.4 

0.1 

1.2 

Two or more industries 

4.2 

# 

t 

t 

T 

Don’t know/refused industry 

1.9 

# 

t 

t 

T 

Not employed 

11.3 

45.8 

4.7 

1.8 

12.2 

CT1. Reporting of the certificate(s) 

Certificate(s) reported 

31.5 

7.5 

(i) 

(i) 

(i) 

No certificate reported 

68.5 

18.5 

2.8 

1.0 

7.7 


t Not applicable 
# Rounds to zero. 

1 Reference group for odds ratio. 

2 Excludes persons of Hispanic ethnicity. 

NOTE: Figures represent unweighted estimates. Observations with missing data for a given variable are excluded. Includes self-reported data only; data completed by proxy were 
excluded from the analysis. Details may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. 
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Table 3-2. Percentage distribution of respondents from the certificates seeded sample frame and percentage and 
odds ratios of seeded sample respondents who did not report the credential listed in the frame 
(underreporting), by selected respondent characteristics: 2010-11 



Percent of respondents 

Percent of respondents who did 
not report the credential listed in 


95 percent confidence 
interval for odds ratio 

Selected respondent characteristics 

from the certificate seeded 
sample frame 

the frame 
(underreporting rate) 

Odds ratio of 
underreporting 

Lower 

bound 

Upper 

bound 

Total 

100.0 

50.0 

(i) 

(i) 

(i) 

ED3. Highest reported level of education 

High school or less than high school 

7.1 

t 

(i) 

(i) 

(i) 

Some college or associate’s degree 

74.3 

50.0 

0.7 

0.1 

4.3 

Bachelor's, graduate, or professional degree 

18.6 

46.2 

0.6 

0.1 

4.6 

ALl.Age 

18 to 34 years old 

46.3 

48.4 

1.0 

0.4 

2.7 

35 to 64 years old 

46.3 

48.4 

(i) 

(i) 

(i) 

65 years old and over 

7.5 

t 

T 

T 

T 

AL3/AL4. Race/ethnicity 

White 2 

51.5 

47.1 

(i) 

(i) 

(i) 

All other races/ethnicities 

48.5 

56.3 

1.4 

0.5 

3.8 

AL6. Sex 

Male 

52.2 

44.4 

(i) 

(i) 

(i) 

Female 

47.8 

57.6 

1.7 

0.6 

4.5 


See notes at end of table. 
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Table 3-2. Percentage distribution of respondents from the certificates seeded sample frame and percentage and 


odds ratios of seeded sample respondents who did not report the credential listed in the frame 
(underreporting), by selected respondent characteristics: 2010-11 — Continued 



Percent of respondents 

Percent of respondents who did 
not report the credential listed in 


95 percent confidence 
interval for odds ratio 

Selected respondent characteristics 

from the certificate seeded 
sample frame 

the frame 
(underreporting rate) 

Odds ratio of 
underreporting 

Lower 

bound 

Upper 

bound 

IN6/IN7. Employment industry 

Private company 

37.7 

46.2 

(i) 

(i) 

(i) 

Government 

14.5 

30.0 

0.5 

0.1 

2.4 

Nonprofit 

10.1 

42.9 

0.9 

0.2 

4.7 

Self-employed 

8.7 

50.0 

1.2 

0.2 

6.9 

Two or more industries 

5.8 

t 

T 

T 

t 

Not employed 

23.2 

62.5 

1.9 

0.5 

6.9 

CN1 . Reporting of the certification(s) 

Certification 

58.6 

51.2 

1.1 

0.4 

2.9 

No certification 

41.4 

48.3 

(i) 

(i) 

(i) 

CN7/CN7A/CN7B. Training and education taken to earn 
most recent certification or license 
Certification received as part of a degree 

45.7 

56.3 

(i) 

(i) 

(i) 

Certification received not as part of a degree 
(includes missing) 

12.9 

33.3 

0.4 

0.1 

1.8 

No certification 

41.4 

48.3 

0.7 

0.3 

2.0 


t Not applicable 

t Data suppressed because reporting standards were not met and estimates are unreliable. 

1 Reference group for odds ratio. 

2 Excludes persons of Hispanic ethnicity. 

NOTE: Figures represent unweighted estimates. Observations with missing data for a given variable are excluded. Includes self-reported data only; data completed by proxy were 
excluded from the analysis. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES), Pilot Study, 2010-11. 
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3.3 Analysis of True Credentials and Credential Commonalities 

Matched cases are considered true credentials because they are a report of a 
credential that is confirmed both in the administrative records data and by the survey data. 
The characteristics of true credentials can provide a rule-of-thumb for identifying reported 
credentials in the nationally representative sample that do not reflect the expected 
characteristics of credentials, as identified using the seeded sample. These cases are 
classified as credential overreports. The following sections provide additional indicators of 
reporting accuracy in the ATES pilot study through the analysis of true credentials and their 
characteristics. 

3.3.1 Matching frame credentials to self-reported credentials 

All of the credentials reported by seeded sample respondents were evaluated to see 
which, if any, matched the seeded sample frame data in order to create a dataset of 
matching and nonmatching seeded credentials. Table 3-3 shows the credential matching 
results for persons selected from the certification and license frames, and table 3-4 shows 
the credential matching results for persons selected from the certificate frame. 20 The overall 
match rate — which takes into account underreports as well as persons with no seeded 
credential matches — was 77 percent for both seeded certifications and seeded licenses, 
yielding a combined overall match rate of 77 percent. 21 This implies that 23 percent of 
certifications and licenses included in the seeded frame were not reported (not shown in 
tables), although error associated with the contacting of seeded sample adults (e.g. the 
wrong adult was reached) could also contribute to this difference. The overall match rate for 
certificates was 41 percent. 22 The relatively low match rate for the certificates largely 
reflects the extensive underreporting among the seeded certificate sample (as discussed 
previously in section 3.2). 


20 For the purposes of this section, responses related to certifications and licenses are reported separately. 

21 The conditional credential match rate — which looks only at the percent of matches where at least one credential of 
the same type was provided during the interview — was 93 percent for the persons sampled from the 
certification/license frame who answered “yes” to CN1. The corresponding conditional credential match rate for 
persons sampled from the license frame was 90 percent, yielding a combined credential match rate of 91 percent. 

2 The conditional credential match rate for the persons sampled from the certificate frame who answered “yes” to 
CT1 was 86 percent. 
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Table 3-3. Percentage of respondents from the certification/license seeded sample 
frame who reported a credential that matched a credential in the frame, 
by credential type and credential matching result: 2010-11 

Total Percent of Percent of 


Comparison of seeded sample survey responses to seeded sample frame percent certifications licenses 


Respondent reported a certification/license in the survey 
Number of reported certifications/licenses with a match in seeded sample frame 

84.5 

82.4 

85.8 

0 

7.8 

5.9 

9.0 

1 

68.9 

64.7 

71.6 

2 or more 

7.8 

11.8 

5.2 

Respondent did not report a certification/license in the survey 

15.5 

17.6 

14.2 

No credentials reported in certification/license or certificate survey sections 

12.8 

14.1 

11.9 

Respondent reported a certificate in the survey 1 

2.7 

3.5 

2.2 

Credential match rate (conditional; person-level) 

90.8 

92.9 

89.6 

Overall match rate (person-level) 

76.7 

76.5 

76.8 


1 1ncludes instances where seeded certification/license was reported as a certificate in the certificate module of the survey as 
well as instances where the certificate reported in the survey did not match the seeded certification/license. 

NOTE: Figures represent unweighted estimates. Includes self-reported data only; data completed by proxy were excluded from 
the analysis. The credential match rate refers to the proportion of reported certifications/licenses matching a certification/license 
in the seeded sample frame. The overall match rate— which takes into account underreports as well as persons with no seeded 
credential matches— refers to the proportion of people on the seeded sample frame who during the survey reported at least one 
of the certifications/licenses provided on the seeded sample frame. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES), 
Pilot Study, 2010-11. 

Table 3-4. Percentage of respondents from the certificates seeded sample frame 
who reported a credential that matched a credential in the frame, by 


credential matching result: 2010-11 

Comparison of seeded sample survey responses to seeded sample frame Percent 

Respondent reported a certificate in the survey 48.0 

Number of reported certificates with a match in seeded sample frame 

0 6.7 

1 37.3 

2 or more matches 4.0 

Respondent did not report a certificate in the survey 52.0 

No credentials reported in certificate or certification/license survey sections 21.3 

Seeded certificate was misreported as a certification/license in the survey 21.3 

Certification/license was reported, but did not match seeded frame certificate 9.3 

See notes at end of table. 
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Table 3-4. Percentage of respondents from the certificates seeded sample frame 
who reported a credential that matched a credential in the frame, by 
credential matching result: 2010-11 — Continued 

Comparison of seeded sample survey responses to seeded sample frame Percent 

Credential match rate (conditional; person-level) 86.1 

Overall match rate (person-level) 41.3 

NOTE: Figures represent unweighted estimates. One certificate sample source was dropped from all analyses because of 
problems with the sample frame. Includes self-reported data only; data completed by proxy were excluded from the analysis. The 
credential match rate refers to the proportion of reported certificates matching a certificate in the seeded sample frame. The 
overall match rate— which takes into account underreports as well as persons with no seeded credential matches— refers to the 
proportion of people on the seeded sample frame who during the survey reported at least one of the certificates provided on the 
seeded sample frame. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES), Pilot Study, 2010-11. 

3.3.2 Items with high rates of common responses in the seeded sample 

Tables 3-5 and 3-6 show the distributions in respondent reports of the certification 
and license-specific items; the distributions are presented for those credentials that 
matched the seeded sample frame — considered to be true credentials and reported in this 
section — by credential type (certification or license). Table 3-5 presents general information 
on all (up to 5) certifications and licenses reported, whereas table 3-6 presents more 
specific information on the most recent certification or license only. Similar to the results 
reported for the national sample in chapter 2, a high percentage of certifications 
(95 percent) that matched the seeded sample frame were obtained for mainly work-related 
reasons (table 3-5). For the most recent matching certification or license reported, 

86 percent of certifications and 68 percent of licenses were reported to be related to the 
respondent’s current job (table 3-6, item CN12). Of these, 60 percent of the certifications 
and 79 percent of the licenses were required for the respondent’s current job (item CN13). 
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Table 3-5. Percentage distribution of true certifications and licenses, by credential 
type and certification/license item: 2010-11 



Percent of 

Percent of 

Certification/license item 1 

certifications 

licenses 


CN4. “Is that a certification, a license, or both?” 

Certification 
License 
Both 

CN6. “Did you get this certification/license mainly for work-related reasons or 
mainly for personal interest?” 

Mainly work-related 
Mainly personal interest 
Don't know/Refused/Missing 
# Rounds to zero. 

t Data suppressed because reporting standards were not met and estimates are unreliable. 

1 1tems asked for each of the first five certifications or licenses, or both. 

NOTE: Figures represent unweighted estimates. True certifications and licenses are credentials reported by respondents that 
match certifications and licenses provided in the sample frames. Includes self-reported data only; data completed by proxy 
were excluded from the analysis. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES), Pilot Study, 2010-11. 


Table 3-6. Percentage of most recent true certifications and licenses, by credential 
type and certification/license item: 2010-11 


Certification/license item 1 

Percent of 
certifications 

Percent of 
licenses 

CN7/CN7A/CN7B/CN7C. Training and education taken to earn certification or 
license 

Took courses or training to earn certification/license 

75.5 

75.0 

Took courses as part of associate's degree, bachelor’s degree, or certificate 
program 

18.4 

10.3 

Took course as part of some other degree program 

57.1 

65.7 

Coursework was mainly self-study 

28.6 

23.5 

Coursework was mainly instructor (nondegree) 

28.6 

41.2 

Did not take courses or training to earn certification/license 

24.5 

25.0 

CN10. “Did you have to do any of the following to get this certification/license...” 

Demonstrate skills while on the job? 

71.4 

62.3 

Pass a test or exam? 

100.0 

79.7 

Submit a portfolio of work? 

55.1 

17.4 

Any of the above 

100.0 

89.9 


See notes at end of table. 


79.2 

9.1 

11.7 


8.1 

67.6 

24.3 


94.8 

5.2 

# 
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Table 3-6. Percentage of most recent true certifications and licenses, by credential 
type and certification/license item: 2010-11 — Continued 


Certification/license item 1 

Percent of 
certifications 

Percent of 
licenses 

CN11. “Did you have to take...” 

Continuing education classes or earn continuing education units to maintain the 

certification/license? 

98.0 

44.9 

Periodic tests? 

28.6 

29.0 

Any of the above 

98.0 

55.1 

CN12. “Is this certification/license related to your current job?” 

Yes 

85.7 

68.1 

CN13. “Is it required for current job?” (Asked only if CN12=Yes) 

Certification/license is related to current job 

100.0 

100.0 

Certification/license is required for current job 

59.5 

78.7 

Certification/license is not required for current job (includes missing) 

40.5 

21.3 

CN14. Were you certified/licensed by your state, industry, a company, a 
professional association, or some other organization? (Mark all that apply.) 

State 

t 

88.4 

Professional association 

71.4 

11.6 

Industry, company, or some other organization 

24.5 

10.1 

Don't know/Refused/Missing 

t 

# 

CN15. “Can this [certification] [license] be...” 

Revoked or suspended for any reason 

81.6 

82.6 

Used to get a job with any employer in that field 

95.9 

100.0 


# Rounds to zero. 

t Data suppressed because reporting standards were not met and estimates are unreliable. 

1 1tems asked of the most recent certifications or licenses, or both. 

NOTE: Figures represent unweighted estimates. True certifications and licenses are credentials reported by respondents that 
match certifications and licenses provided in the seeded sample frames. Includes self-reported data only; data completed by 
proxy were excluded from the analysis. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES), Pilot Study, 2010-11. 

Among the matching seeded sample credentials, 76 percent of the most recent 
certifications and 75 percent of the most recent licenses reported were earned after taking 
some type of courses and/or training (table 3-6, item CN7). In addition, 1 00 percent of the 
most recent matching certifications and 80 percent of the most recent licenses required 
respondents to pass a test or exam; 71 percent of certifications and 62 percent of licenses 
required a demonstration of skills while on the job; and 55 percent of certifications and 
17 percent of licenses required the submission of a portfolio of work (item CN10). Nearly all 
certifications and licenses had at least one of these requirements (100 percent of 
certifications and 90 percent of licenses). 
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After the most recent matching certifications or licenses were obtained, 98 and 
45 percent of these credentials, respectively, required respondents to take continuing 
education units to maintain them (table 3-6, item CN11). The majority of the certifications 
were issued by a professional association (71 percent), while most of the licenses were 
issued by a state (88 percent, item CN14). According to respondents, 82 percent of the 
certifications and 83 percent of the licenses could be revoked or suspended, and 96 percent 
of the certifications and 100 percent of the licenses could be used to get a job with any 
employer in the field (item CN15). 

The factors associated with certificate matches also varied in expected ways. 
Community colleges were the most frequently cited source of the most recent matching 
certificate (>80 percent), followed by university or college and a trade school (not shown in 
tables). The median number of credit hours required for the most recent matching certificate 
was 27, although it should be noted that 36 percent of certificates required less than 
17 credits to complete (not shown in tables). 

3.3.3 Using credential commonalities to estimate overreporting in the national 
sample 

The next analysis considers whether the credential commonalities that were 
identified using the seeded sample can be used to identify possible instances of 
overreporting of credentials in the national sample (“false positives”), by determining the 
extent to which a reported certification/license does not have the characteristics of a typical 
credential. Using this approach, uncommon responses (e.g., reporting a certification/ 
license, but not reporting that the certification/license was work-related) are used to flag 
possible instances of overreporting. 

Table 3-7 shows the percentage of matching seeded sample certifications/licenses 
that would be miscategorized as an overreport in the seeded sample — and possibly in the 
national sample — based on uncommon responses to various combinations of three related 
factors measured by items CN6 (credential for personal interest), CNIOa/b (no test or 
demonstration of skills required), and CN15b (credential could not be used to get a job). 

The results of the seeded sample, which is limited in its coverage of credentials, suggests 
that all combinations perform fairly well at identifying true positives (while minimizing 
overreporting) and that any one of the factors could be used independently to eliminate 
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overreporting. It should be noted that because the sample was not representative of all 
types of credential holders, it is possible that some uncommon responses were not 
misreports. The national sample results provide some indication of how many 
certifications/licenses would be identified as overreports using the various criteria 
combinations. While comparison record data are not available for the national sample, 
those results imply that while there is some overlap between CNIOa/b (no test or 
demonstration of skills required) and CN15b (credential could not be used to get a job), 
using CN6 (credential for person of interest) alone or with CNIOa/b and/or CN15b would 
identify a slightly different group of possible certification/license overreports. 23 

3.4 Proxy Responses to Certification/License and Certificate Items 

To determine whether proxy respondents are a reliable source of data on 
certifications/licenses and certificates, proxy reported data were compared to self-reported 
data. 24 Information was collected from a proxy responder for about 1 ,000 households. The 
proxy responder was an adult in the household who answered the survey questions about 
another adult in the household. A self-responder answered questions about him or herself. 

The study did not include a proxy report validation. The intended focus of the proxy 
study was to analyze the percentage of “don’t know” responses in the proxy-reported 
responses compared to self-reported responses. For most questions in the survey, a 
response of “don’t know” would be considered item-missing data. High rates of item-missing 
data can potentially lead to bias in the survey estimates. Therefore, first, the frequency of 
“don’t know” responses to each certification/license and certificate item was examined for 
proxy and self-reported data to provide an indication of the extent to which proxy 
respondents, in comparison to self-reporting respondents, are able to provide the requested 
information (tables 3-8 and 3-9). Second, the analysis looked at differences in estimates of 
valid responses when “don’t know” is excluded as an indication of potential bias in proxy 
reporting. Estimates that exclude “don’t know” are presented in table C-19 in appendix C. 


23 This analysis was not performed for the certificates items because of other measurement issues 
reported previously (e.g., a high underreporting rate on the main certificate item). 

24 See appendix B for more information about the proxy respondents included in the ATES Pilot Study. 
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The results of the analysis suggest that proxy respondents were a reasonable 
source of information on the main credential items and on some of the additional items, but 
not on others. Items that prompt respondents for detailed information — such as a credential 
name, an estimate of the year earned, or time to complete a credential — were particularly 
difficult for proxy respondents. 


Table 3-7. Percentage of true certifications/licenses that would be miscategorized 
as overreports, by type of certification/license sample and criteria for 
identifying overreports: 2010-11 


Criteria for identifying overreports 

Percent 
reported by the 
seeded sample 

Percent 
reported by the 
national sample 

Based on one item asked of certification/license holders... 
Certification/license was pursued for personal interest (item CN6) 

t 

6.4 

Certification/license did not require a demonstration of skills while on the job 
or passing a test or exam (item CNIOa/b) 

5.1 

4.5 

Certification/license could not be used to get a job with any employer in that 
field (item CN1 5b) 

# 

4.2 

Based on two items asked of certification/license holders... 

Certification/license was pursued for personal interest (item CN6) OR 
certification/license did not require a demonstration of skills while on the job 
or passing a test or exam (item lOa/b) 

6.8 

10.5 

Certification/license was pursued for personal interest (item CN6) OR 
certification/license could not be used to get a job with any employer in that 
field (item CN1 5b) 

t 

10.3 

Certification/license did not require a demonstration of skills while on the job 
or passing a test or exam (item lOa/b) OR certification/license could not be 
used to get a job with any employer in that field (item CN1 5b) 

5.1 

7.6 

Based on three items asked of certification/license holders... 

Certification/license was pursued for personal interest (item CN6) OR 
certification/license did not require a demonstration of skills while on the job 
or passing a test or exam (item lOa/b) OR certification/license could be 
used to get a job with any employer in that field (item 1 5b) 

6.8 

13.3 


# Rounds to zero. 

$ Data suppressed because reporting standards were not met and estimates are unreliable. 

NOTE: Figures represent unweighted estimates. True certifications/licenses are credentials reported by 
respondents that match certifications/licenses provided in the sample frames. Data from all 
certification/license questions were available for 1 19 certifications/licenses that matched seeded sample 
frame credentials, and 1,229 certifications/licenses from the national sample. “Don’t know” responses 
and refusals were treated as neutral. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES), 
Pilot Study, 2010-11. 
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Table 3-8. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and selected survey items: 2010-11 


Selected survey items 

Reported by 
sampled adult 
(self-reported) 

Reported by 
another household 
member on behalf 
of sampled adult 
(proxy-reported) 

Percent difference 1 

CN1. “Now I'd like to ask you about professional 
certification and licensure. Do you/person have a 
professional certification or a state or industry 
license?” 

Yes 

32.9 

30.4 

-2.5 

No 

66.6 

62.9 

-3.7 

Don’t know 

# 

6.6 

t 

CN1A. “Have more than one certification or license?” 

Yes 

34.9 

29.3 

-5.6 

No 

64.7 

62.2 

-2.5 

Don't know 

# 

8.5 

t 

CN1 B. “How many certifications/licenses do you/person 
have?” 

1 

64.8 

62.2 

-2.6 

2 

21.2 

17.1 

-4.1 

3 

7.9 

5.5 

-2.4 

4 

2.8 

1.5 

-1.3 

5 or more 

2.7 

0.9 

-1.8 

Don't know 

0.7 

12.8 

12.1 

CN3AR1. “Name of most recent certification/license” 

Name reported 

98.6 

86.8 

-11.8 

Don't know 

1.4 

13.2 

11.8 

CN4R1. “Is that a certification, a license, or both?” 

Certification 

41.3 

32.2 

-9.1 

License 

33.6 

27.6 

-6.0 

Both 

22.7 

26.4 

3.7 

Don't know 

2.4 

13.8 

11.4 


See notes at end of table. 
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Table 3-8. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and select survey items: 2010-11 — Continued 


Select survey items 

Reported by 
sampled adult 
(self-reported) 

Reported by 
another household 
member on behalf 
of sampled adult 
(proxy-reported) 

Percent difference 1 

CN6R1. “Did you/person get this certification/license 
mainly for work-related reasons, or mainly for personal 
interest?” 

Mainly work related 

91.6 

91.0 

-0.6 

Mainly personal interest 

7.9 

7.5 

-0.5 

Don't know 

# 

1.6 

t 

CN7A. “Did you/person take the coursework or training 
for the certification/license as part of a college or 
university certificate or degree program?” 

Yes 

55.3 

55.8 

0.5 

No 

43.7 

36.6 

-7.1 

Don't know 

1.0 

7.5 

6.5 

CN7B. “Was that a certificate program, an associate's 
degree program, a bachelor's degree program, or 
something else?” 

Certificate 

18.3 

10.4 

-7.9 

Associate's 

10.4 

11.0 

0.6 

Bachelor's 

33.8 

40.5 

6.7 

Other 

35.1 

32.5 

-2.6 

Don't know 

2.4 

5.5 

3.2 

CN7C. “Was the coursework or training mainly 
self-study or mainly classes or courses with an 
instructor?” 

Mainly self-study 

11.8 

11.0 

-0.8 

Mainly with and instructor 

87.6 

85.6 

-2.0 

Don't know 

0.6 

3.4 

2.9 

CN8NR. “How many weeks or months was the course?” 
Answer reported 

93.0 

65.5 

-27.5 

Don't know 

7.0 

34.5 

27.5 

CN9NR. “How many hours with an instructor?” 
Answer reported 

88.2 

50.7 

-37.4 

Don't know 

11.8 

49.3 

37.4 


See notes at end of table. 
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Table 3-8. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and select survey items: 2010-11 — Continued 


Select survey items 

Reported by 
sampled adult 
(self-reported) 

Reported by 
another household 
member on behalf 
of sampled adult 
(proxy-reported) 

Percent difference 1 

“Did you/person have to do any of the following 
to get this certification/license...” 

CN10A. demonstrate skills while on-the-job?” 

Yes 

65.7 

65.5 

-0.2 

No 

33.6 

24.5 

-9.2 

Don't know 

0.7 

10.0 

9.4 

CN10B. “- pass a test or exam?” 

Yes 

91.0 

90.6 

-0.4 

No 

t 

5.6 

t 

Don't know 

# 

3.8 

t 

CN10C. “...submit a portfolio of work?” 

Yes 

31.4 

26.6 

-4.8 

No 

66.0 

53.6 

-12.4 

Don't know 

2.6 

19.7 

17.2 

“Have to take...” 

CN1 1A. continuing education classes or 

earn continuing education units to maintain it?” 

Yes 

63.6 

58.0 

-5.6 

No 

35.3 

33.2 

-2.1 

Don't know 

1.1 

8.8 

7.7 

CN11B. periodic tests?” 

Yes 

31.0 

32.0 

1.0 

No 

67.2 

54.2 

-13.0 

Don't know 

1.8 

13.8 

12.0 

CN12. “Is this certification related to current job?” 

Yes 

81.9 

85.6 

3.7 

No 

t 

14.4 

t 

Don't know 

# 

# 

t 

CN13. “Is it required for current job?” 

Yes 

72.7 

79.9 

7.2 

No 

4 

16.9 

t 

Don't know 

# 

3.2 

t 


See notes at end of table. 
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Table 3-8. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and select survey items: 2010-11 — Continued 


Select survey items 

Reported by 
sampled adult 
(self-reported) 

Reported by 
another household 
member on behalf 
of sampled adult 
(proxy-reported) 

Percent difference 1 

CN14. “[certified] [licensed] by ...” 

State 

67.8 

68.9 

1.1 

Industry 

2.6 

3.8 

1.2 

Company 

5.1 

2.5 

-2.6 

Professional association 

15.9 

10.4 

-5.5 

Some other organization 

13.8 

9.1 

-4.7 

Don't know 

1.6 

9.7 

8.2 

“Can this certification/license be...” 

CN15A. “- revoked or suspended for any reason?” 

Yes 

76.3 

66.9 

-9.4 

No 

14.8 

13.2 

-1.6 

Don't know 

8.9 

19.9 

10.9 

CN15B. used if you/person wanted to get a job with 
any employer in that field?” 

Yes 

92.9 

91.8 

-1.1 

No 

4.3 

1.9 

-2.5 

Don't know 

2.8 

6.3 

3.5 

CN17. “Is the certification/license currently valid or has it 
expired?” 

Currently valid 

70.7 

66.1 

-4.7 

Expired 

24.1 

26.7 

2.5 

Don't know 

5.1 

7.3 

2.2 

CN18. “Do you/person plan to renew it?” 

Yes 

11.2 

t 

T 

No 

85.9 

t 

T 

Don't know 

2.9 

# 

T 


# Rounds to zero, 
t Not applicable. 

t Data suppressed because reporting standards were not met and estimates are unreliable. 

1 Difference is the proxy percentage minus the self-reported percentage. 

NOTE: Figures represent unweighted estimates. Only valid responses are reported in the table; respondents not asked 
questions because of legitimate skips due to earlier responses are excluded. Item level refusals are also excluded from this table 
because in all instances the sample sizes were too small to meet reporting standards. 

Source: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES), 
Pilot Study, 2010-11. 
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Table 3-9. Percentage distribution of responses to certificate survey items, by 

reporting method, difference between the reporting method, and select 
survey items: 2010-11 


Select survey items 

Reported by 
sampled adult 
(self-reported) 

Reported by another 
household member 
on behalf of sampled 
adult (proxy-reported) 

Percent 

difference 1 

CT 1 . “Some people decide to enroll at a college, 
university, community college, or trade school to 
earn a certificate rather than a degree. Have 
you/person ever earned this type of certificate?” 

Yes 

14.2 

9.1 

-5.1 

No 

85.4 

84.0 

-1.4 

Don't know 

# 

6.9 

t 

CT2. “Do you/person have more than one certificate?” 

Yes 

29.7 

21.6 

-8.0 

No 

t 

71.1 

t 

Don't know 

# 

7.2 

t 

CT2A. “How many certificates do you/person have?” 

1 

70.1 

71.1 

1.0 

2 

15.0 

8.2 

-6.7 

3 

7.2 

5.2 

-2.0 

4 or more 

6.2 

3.1 

-2.9 

Don't know 

1.6 

12.4 

10.8 

CT2CR1 . “Is most recent certificate part of the 
coursework or training you/person took for a 
certification or license?” 

Yes 

72.5 

85.0 

12.5 

No 

24.8 

15.0 

t 

Don't know 

2.8 

# 

t 

CT3R1 . “What is the subject or major field of study?” 

Answer reported 

98.8 

90.7 

-8.1 

Don't know 

1.2 

9.3 

8.1 

CT4R1 . “What is the name of the certificate?” 

Answer reported 

82.1 

52.6 

-29.5 

Don't know 

17.9 

47.4 

29.5 

CT6R1. “In what year did you/person complete it?” 

Answer reported 

93.3 

69.6 

-23.8 

Don't know 

6.7 

30.4 

23.8 

CT8. “What type of school or organization provided the 
certificate program?” 

Community college 

20.8 

22.9 

2.1 


See notes at end of table. 
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Table 3-9. Percentage distribution of responses to certificate survey items, by 

reporting method, difference between the reporting method, and select 
survey items: 2010-11 — Continued 


Select survey items 

Reported by 
sampled adult 
(self-reported) 

Reported by another 
household member 
on behalf of sampled 
adult (proxy-reported) 

Percent 

difference 1 

CT8— Continued 

Another university or college 

14.9 

13.5 

-1.4 

Trade school 

21.8 

24.0 

2.2 

Business or company 

5.3 

4.2 

-1.1 

Professional association 

2.9 

t 

T 

Trade union 

1.6 

t 

T 

Nonprofit organization 

0.8 

# 

T 

Federal, state, local government 

3.5 

t 

t 

Other 

25.9 

16.7 

-9.3 

Don't know 

2.4 

14.6 

12.2 

CT9. “How many credits or hours did it take?” 

Answer reported 

45.4 

30.9 

-14.5 

Don't know 

54.6 

69.1 

14.5 

CT10. “Did you/person take the education or training to 
earn continuing education units or continuing 
education units?” 

Yes 

27.3 

21.1 

-6.3 

No 

64.8 

58.9 

-5.9 

Don't know 

7.9 

20.0 

12.1 

CT1 1 . “Was the training mainly self-study or mainly 
classes or courses with an instructor?” 

Mainly self-study 

9.9 

4.8 

-5.1 

Mainly with and instructor 

87.8 

88.7 

0.9 

Don't know 

2.3 

6.5 

4.2 

CT12. “Is this certificate related to your/person current 
job?” 

Yes 

49.4 

50.0 

0.6 

No 

t 

t 

T 

Don't know 

t 

t 

t 


See notes at end of table. 


48 






Table 3-9. Percentage distribution of responses to certificate survey items, by 

reporting method, difference between the reporting method, and select 
survey items: 2010-11 — Continued 


Select survey items 

Reported by 
sampled adult 
(self-reported) 

Reported by another 
household member 
on behalf of sampled 
adult (proxy-reported) 

Percent 

difference 1 

CT1 3. “Is it required for current job?” 




Yes 

26.2 

47.1 

20.9 

No 

73.8 

t 

T 

Don't know 

# 

t 

T 


# Rounds to zero, 
t Not applicable. 

t Data suppressed because reporting standards were not met and estimates are unreliable. 

1 Difference is the proxy percentage minus the self-reported percentage. 

NOTE: Figures represent unweighted estimates. Only valid responses are reported in the table; respondents not asked 
questions because of legitimate skips due to earlier responses are excluded. Item level refusals are also excluded from this table 
because in all instances the sample sizes were too small to meet reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES), 
Pilot Study, 2010-11. 

3.5 Use of Help Text Probes 

The ATES Pilot Study questionnaire offered interviewers a standardized probe that 
could be used if respondents’ asked for further explanation or expressed confusion about 
the main certification/license and certificate items. Interviewers recorded when they read the 
probe to the respondent. The estimate of the main certifications/licenses item when the 
certification/license probe was used was similar to the estimate when it was not used. This 
suggests that the probe confirmed respondents’ interpretation of the meaning of the item. 
However, the use of the certificates probe approximately doubled reports of certificates. 


The probe for certifications/licenses was: “A professional certification or license 
shows you are qualified to perform a specific job and includes things like Licensed Realtor, 
Certified Medical Assistant, Certified Construction Manager, a Project Management 
Professional or PMP certification, or an IT certification.” 


The probe for certifications/licenses was used for 17 percent of all respondents; 

34 percent of respondents who received the probe reported they held a certification/license 
compared to 33 percent of respondents who did not receive the probe (unweighted 
estimates; see table C-16 in appendix C). 
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The probe for certificates was: “An educational certificate is typically earned by 
completing a program of study offered by a college or university, a community college, or a 
trade school, but it does not lead to an associate’s, bachelor’s, or graduate degree. 
Sometimes these are also called vocational diplomas, for example, a cosmetology or 
mechanics diploma, which differs from a high school diploma.” 

The probe for certificates was used for 15 percent of all respondents; 27 percent of 
respondents who received the probe reported they had earned a certificate compared to 
12 percent of respondents who did not receive the probe (unweighted estimates; see 
table C-17 in appendix C). 

3.6 Summary of Findings on Measurement Error 

This chapter presented findings on measurement error in the ATES Pilot Study, 
including an analysis of the underreporting of credentials, credential commonalities among 
true positives (defining characteristics of credentials), and proxy responses. The analysis of 
underreporting used data from the seeded certification, license, and certificate samples to 
calculate the percentage of known credentials that respondents did not report during the 
study. The overall underreporting rate for seeded certifications/licenses was found to be 
15 percent, while the underreporting rate for seeded certificates was 50 percent. These 
rates did not vary by most subgroups; however, multivariate analyses suggested that the 
rates are being driven, in part, by labor force status and that employed respondents were 
more accurate in their reporting of seeded credentials than were other respondents. This 
finding may reflect credential “salience;” because most credentials are work-related, 
respondents who are working view their credentials as relevant and therefore report them. 
Respondents who are not working do not attach as much value to credentials they are not 
using, and therefore do not report them. However, this possibility was not addressed in the 
current study. 

Credential commonalities were analyzed for the “true positive” credentials — that is, 
reported credentials that matched those in the seeded sample frames. This analysis found 
that the characteristics expected to be related to certifications/licenses and certificates did, 
in fact, covary with reports of those credentials in predictable ways. Similar to the results 
reported in chapter 2 for the national sample, a high percentage of certifications 
(95 percent) that matched the seeded sample frame were obtained mainly for work-related 
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reasons. In addition, 100 percent of the most recent matching certifications and 80 percent 
of the most recent matching licenses required respondents to pass a test or exam; 

96 percent of these certifications and 100 percent of the licenses could be used to get a 
job with any employer in the field. These results suggest that these three characteristics 
(measured by items CN6, CNIOb, and CN15b) can be considered “defining characteristics” 
of certifications and licenses, and can therefore be used to identify overreporting of these 
credentials. 

The factors associated with certificate true positives also varied in expected ways. 
Community colleges were the most frequently cited source of the most recent matching 
certificate (81 percent), followed by another university or college (12 percent) and a trade 
school (8 percent). The median number of credit hours required for the most recent 
matching certificate was 27, although it should be noted that 36 percent of certificates 
required less than 17 credits to complete. 

Next, this chapter presented the results of an analysis of proxy respondents’ 
responses. To determine whether proxy respondents are a reliable source of data on 
certifications/licenses and certificates, the frequency of proxy respondents’ responses to 
each certification/license and certificate item was compared to the frequency of self- 
reported responses. The results of the analysis suggested that proxy respondents were a 
reasonable source of information on the main credential items and on some of the related 
items — such as whether the credential was obtained for work-related reasons — but not a 
good source of information on some of the other items. Items that prompt respondents for 
detailed information — such as a credential name, an estimate of the year earned, or the 
time to complete a credential — were particularly difficult for proxy respondents. 

Finally, an analysis of the use of help text probes showed that comparable numbers 
of respondents reported a certification or license after hearing the help text for the main 
certification item but that more respondents reported a certificate after hearing the help text 
for the main certificate item. This may suggest that the meaning of the main 
certification/license item is clear without the help text while the main certificate item requires 
additional clarification. 
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4. Conclusions and Next Steps 

The primary goal of the ATES Pilot Study was to identify a parsimonious set of 
survey items that can be used to generate valid population estimates of the prevalence of 
certifications/licenses and certificates in the U.S. adult population. To evaluate survey items, 
this report examined two topics: 

1 . Do the population estimates generated from the ATES Pilot Study measures seem 
reasonable? 

2. To what extent do respondents misreport certifications, licenses, and certificates, 
and what are the characteristics of respondents and credentials associated with 
misreports? To what extent can proxy respondents provide valid answers to 
questions about certifications, licenses, and certificates? 

This chapter first provides a summary of the answers to each of these questions. 
Next, the chapter presents GEMEnA recommendations — based on the ATES Pilot Study 
results and GEMEnA’s subsequent survey development work — for survey items appropriate 
for addressing researchers’ and policymakers’ concerns about industry-recognized 
certifications, occupational licenses and educational certificates. Finally, the chapter 
provides an overview of future work GEMEnA member agencies hope to pursue in order to 
refine and expand the data collected by the federal statistics system on the role of these 
credentials in signaling labor force readiness and on the economic returns to individual 
credential attainment. 

4.1 Summary of Findings From the ATES Pilot Study 

4.1.1 Do the ATES Pilot Study estimates seem reasonable? 

As shown in chapter 2, the ATES estimates of the overall rates of 
certifications/licenses and certificates among U.S. adults were reasonable when compared 
to estimates from other surveys of similar populations, with differences of 3 to 4 percentage 
points. Based on similar samples of currently employed adults, the percentage of adults 
reporting a certification/license was 38 percent in ATES and 34 percent in the Princeton 
Data Improvement Initiative (PDII) sample. In addition, using similar samples of adults ages 
23-64 who reported an income or reported working for earnings in the past year, the overall 
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rate of certificates was 15 percent based on ATES and 18 percent based on the Survey of 
Income and Program Participation (SIPP). These differences were statistically significant; 
however, given that the surveys were administered in different years and had different item 
wording, a difference of 3 to 4 percentage points can be considered reasonable. 

The results from credential-specific items were mixed. Most items examined showed 
the expected results and were therefore considered reasonable. For example, 90 percent of 
certifications/licenses were pursued for mainly work-related reasons, and 91 percent of the 
most recent work-related certifications/licenses required respondents to pass a test or 
exam. However, certain types of items were problematic. For example, it was difficult for 
respondents to report on the time spent in coursework or training pursuing a credential. The 
reported number of hours required to attain a certificate varied widely, and over half of 
certificate holders reported that they did not know the required hours. 

In addition, 26 percent of certificates were reported as coming from “other” sources 
than educational institutions. An analysis of the write-in responses to this item showed that 
half (or 13 percent of “other” source responses) of the 26 percent actually were conferred 
from an educational institution, suggesting a problem with item response options or 
interviewer coding (not shown in tables). An analysis of “true positives” (certificates listed in 
the seeded sample and later reported during the survey) offers another possibility for the 
remaining 13 percent of cases. This analysis found that among confirmed certificates, no 
respondents reported obtaining their credential from “other” sources. The credentials held 
by seeded sample respondents were not representative of all types of certifications; 
however, this finding suggests that a subset of the national sample reports of “other” 
sources of certificates may actually be overreports (noncertificate holders). 

4.1.2 To what extent do respondents misreport certifications, licenses, and 
certificates, and what are the characteristics of respondents and credentials 
associated with misreports? To what extent can proxy respondents provide valid 
answers to questions about certifications, licenses, and certificates? 

Chapter 3 presented overall findings on measurement error in the ATES Pilot Study. 
In the seeded sample certifications/licenses were underreported by 15 percent, while 
certificates were underreported by 50 percent. Multivariate analyses suggested that the 
underreporting was being driven, in part, by labor force status and that employed 
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respondents were more accurate in their reporting than were respondents who were not 
employed. As with all results based on the seeded sample (a nonrepresentative 
convenience sample), these results should be considered illustrative rather than definitive. 
The finding on certificates is particularly tentative; issues with one of the seeded sampling 
frames raised the possibility that the remaining certificate frames contained individuals who 
did not, in fact, hold the credential defined for this study as an educational certificate, in 
which case at least some of what appears to be underreporting may be accurate reporting. 

To determine whether proxy respondents could reliably report on 
certifications/licenses and certificates, chapter 3 included a comparison of proxy and 
self-reported responses to the main certification/license and certificate items and to the 
credential-specific items. The results of the analysis suggested that proxy respondents were 
a reasonable source of information on the main credential items and on some of the related 
items — such as whether the credential was obtained for work-related reasons — but not a 
good source of information on some of the other items. Items that prompt respondents for 
detailed information — such as a credential name, the year a credential was earned, or the 
time to complete a credential — were particularly difficult for proxy respondents. 

4.2 Recommendations for Items to Include in Other Surveys 

The ATES Pilot Study was conducted as part of a multiyear, interagency effort to 
improve the measurement of nondegree credentials in federal household studies of adults. 
The findings in this report have implications for determining a parsimonious set of items that 
could be included in household surveys of adults — either in the near future for those items 
requiring no further development or after additional testing has been conducted to address 
the measurement issues identified in this report. It is important to note that the items 
developed for the pilot test were limited in their scope. They were designed specifically to 
enumerate adults with specific nondegree credentials, and their utility as survey measures 
outside the scope of this Pilot Study has not been tested. Therefore, although results of the 
pilot are used in this section to recommend items for researchers with various goals, the 
properties of the items may differ in other survey administration contexts. 
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With that caveat, the remainder of this chapter summarizes GEMEnA’s 
recommendations for items to use in other household surveys trying to achieve the following 
research goals: 

• Counting individuals with a certification or license; 

• Distinguishing between certifications and licenses; 

• Describing the field of certification or license; and 

• Determining whether the certification or license is related to work. 

The recommendations of GEMEnA are based on the results of the ATES Pilot Study, 
described in this report, as well as GEMEnA’s subsequent survey development work 
conducted in preparation for a related study, the 2013 National Adult Training and 
Education Survey (NATES) Pilot Study. 25 This section ends with a discussion of the main 
measurement lessons from the ATES Pilot Study. 

4.2.1 Counting individuals with a certification or license 

To count the number of individuals in the United States who have a certification or 
license, GEMEnA recommends pilot item CN1: 

Now I'd like to ask you about professional certification and licensure. Do 

you/person have a professional certification or a state or industry license? 

This recommended item is referred to as the main certification/license item; it is an 
aggregate, broad measurement item that could be administered alone or in conjunction with 
credential-specific items to provide counts of adults with work-relevant credentials. To give 
analysts the option to reduce measurement error, survey sponsors could adopt one to three 
additional items that the seeded sample results suggest describe the characteristics of true 
certifications and licenses. These items include whether or not the certification/license is 


25 The subsequent survey development work consisted of 14 cognitive interviews conducted in preparation for the 
2013 NATES Pilot Study, as well as additional focus groups of certificate holders — many of whom also held 
certifications and licenses — conducted to inform future potential administrations of the NATES. The 2013 NATES 
Pilot Study is a household survey of adults being conducted to test the feasibility of administering an education and 
training survey by mail rather than by telephone. More information on the NATES is available on the GEMEnA 
website, at http://nces.ed.gov/survevs/qemena/strand4.asp . 
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work-related (CN6); if the individual had to pass a test or exam (CN1 Ob); and if the 
certification/license is transferable (CN15b). When considering these additional items, 
however, analysts should recognize their potential for eliminating true credentials if used as 
screening questions. Not all certifications and licenses require a test or exam, for example. 

The main certificate item was found to have a high underreporting rate among the 
seeded sample respondents in the ATES Pilot Study and is not recommended for use at 
this time. The main certificate item and the series of items tested in the certificate 
questionnaire section require additional testing before they can be used. This ongoing 
development work is the focus of GEMEnA work described in more detail in Section 4.3. 

4.2.2 Distinguishing between certifications and licenses 

Researchers and policymakers have informed GEMEnA that it is important to 
distinguish between industry-recognized certification and occupational licenses. Because 
certifications are awarded by a credentialing body while licenses are issued by a 
government (often the state), these two types of credentials may be subject to different 
policy levers. Because certifications reflect a demonstration of skill while licenses convey a 
legal authority to work in an occupation, the questions researchers ask about these 
two types of credentials may differ. The ATES Pilot Study asked a follow-up question 
(item CN4) to the main certification/license question to see if the respondent considered a 
credential a certification, a license, or both. 

However, the focus group, cognitive interview, and ATES Pilot Study findings, as 
well as GEMEnA’s subsequent survey development work, confirmed that some respondents 
have difficulty distinguishing between certifications and licenses, particularly in occupations 
(including many healthcare fields) in which an industry-recognized certification 
demonstrating skill attainment is the main requirement for the state license. Therefore, 
GEMEnA recommends the inclusion of an item designed to help determine whether a 
respondent holds a certification or a license, rather than rely on respondents’ direct report 
using item CN4. 

One of the defining differences in the definitions of certifications and licenses in the 
ATES Pilot Study was the entity that awarded the credential. Certifications are awarded by 
industry or professional organizations while licenses are issued by government agencies. 
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The ATES pilot study asked the respondent what kind of organization awarded the 
credential with item CN14, which asked respondents whether they were certified/licensed 
by their state, industry, a company, a professional association, or some other organization. 

However, based on the ATES findings and GEMEnA’s subsequent survey 
development work, GEMEnA recommends the following question be used instead of ATES 
item CN14 to distinguish between certifications and licenses: 

Who issued this certification or license? (Mark one.) 

□ Federal, state, or local government 

□ Professional or trade association (for example, Pediatric Nursing Certification 
Board, National Exercise and Sports Trainers Association, CompTIA) 

□ Business or company (for example, Microsoft, 3M Company, Xerox) 

□ Other group or organization (specify) 


r 

A 

V 

J 


Analysts could code a response to the first option as a license and responses to the 
second and third options as certifications. The other (specify) category could be post-coded 
(as possible) and used to assess the inclusivity of the response options, particularly in a 
pilot study context. This question will be fielded in the 2013 NATES Pilot Study, described in 
more detail in Section 4.3. Because certification requirements for licensure vary by 
occupation and state, surveys focusing on the distinction between these types of credentials 
may want to include additional items to add more specificity. 

4.2.3 Describing the field of certification or license 

GEMEnA recognizes that information about the field, industry, or occupation of a 
credential is critical to answering important questions about the relationship between 
employer needs and the supply of human capital. The dynamic of credential supply and 
demand takes place within specific occupational fields. National household studies 
measuring certifications and licenses should consider including questions to identify the 
field of the credential. The ATES Pilot Study asked respondents for verbatim responses to a 
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question about the name of the certification or license (CN3A) and the kind of work it was 
for (CN3). Based on the ATES findings and GEMEnA’s subsequent survey development 
work, GEMEnA recommends the following questions be used instead of ATES items CN3A 
and CN3: 


What is the name of your [TEXT1][MOST RECENT] certification or license? 
Please do not use abbreviations. 


Write in: 


What kind of work is this certification or license for? (For example: teaching, 
vocational nursing, computer network administration, auditing, truck driving) 

f 

v 


Surveys with limited space can limit these questions to respondents’ most recent 
certification or license, while surveys with more space can collect information on multiple 
credentials of this type. The responses to these questions can be collected verbatim and 
used individually, or used together to develop a set of occupation categories after data 
collection. The second question gives coders additional information to clarify the response. 

4.2.4 Determining if the certification or license is related to work 

GEMEnA has heard from the Council of Economic Advisors, prominent researchers, 
and other experts about the importance of specifying whether a credential is related to the 
respondent’s current job. The question used in the ATES Pilot Study (item CN6) focused on 
whether or not the credential was primarily “work-related” and, was found to be highly 
associated with true certifications and licenses in the seeded sample. However, based on 
GEMEnA’s preference to focus the item on the respondent’s current job, the following item 
was tested in cognitive interviews and included in the 2013 NATES Pilot Study: 

Is this certification or license for the job you have now? If you are currently not 
employed, please answer “no.” (Mark one.) 
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□ No 

□ Yes, and it is required for my job 

□ Yes, but it is NOT required for my job 


GEMEnA recommends this adaptation of the ATES question to clarify the 
relationship between the credential and the current job and to distinguish between 
credentials required for the job and those related to, but not required for, the job. Household 
surveys conducted by telephone or computer-assisted personal interviews may need to 
adapt this item for oral administration to make clear to the respondent the need to 
distinguish between credentials required and not required for the job. 

4.2.5 Measurement implications for certifications/licenses 

The results of the pilot have several implications for measurement. First, the items in 
the ATES were tested in a short survey focused on certifications, licenses, and certificates. 
It is possible that the items may perform differently in different survey contexts, particularly 
in surveys where other topics are covered in more depth. In the ATES Pilot Study context, 
additional measurement implications are discussed below. 

Cognitive testing for the ATES showed an order effect when asking about 
certifications and certificates. If certification holders were asked about certificates before 
certifications, they would report a certification as a certificate. Also, cognitive testing 
confirmed that respondents do not think of certifications and licenses as educational 
attainment, leading the ATES to ask about them in separate sections. Researchers should 
carefully consider the placement of the items within a survey. 

As mentioned previously, the salience of the credential (as characterized by time 
and relevance) affects the ability to collect even basic information such as date earned and 
credential name. It also affects which credentials are reported. If respondents are reporting 
their most salient credentials, results may be biased toward the credentials held and used 
by those currently employed. 

The ATES collected limited detailed information about each specific credential and 
focused the majority of detailed items on the most recent credential. While it is possible to 
ask many questions about each certification or license, it is important to consider that 
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(1) few respondents reported more than 2 credentials; (2) it takes additional time for the 
respondent to report detail about multiple certifications; and (3) the reporting error 
associated with older and less salient credentials could be high. Additionally, proxy 
reporting exacerbates problems with items asking for detailed information. 

When considering sampling options, it is important to note that respondents who 
were not employed had greater reporting error compared to employed respondents. 

4.3 Next Steps on Education, Training, and Credentials for Work 

GEMEnA has defined a multiyear program of rigorous item development to improve 
how federal statistical agencies gather information about education, training, and 
credentials for work. Building from the initial work described in this report, GEMEnA’s 
current data development program is summarized in the four strands of work described in 
detail below. 

4.3.1 Strand 1: Certifications and Licenses 

GEMEnA’s first strand of work is to develop and deploy a core set of survey Items 
related to the prevalence and key characteristics of Industry-recognized certifications and 
occupational licenses. The ATES pilot test described in this report was the first step in this 
strand of work. Moving forward, GEMEnA member agencies are looking within their own 
agencies for opportunities to incorporate the ATES-tested items into ongoing or planned 
studies of out-of-school youth and adults. In addition, they are seeking ways to test these 
items further to explore their performance in different survey contexts. These plans are 
described below. 

NCES. NCES is incorporating a set of questions on certifications/licenses into its 
upcoming Baccalaureate and Beyond (B&B) longitudinal study of 2008 Bachelor’s degree 
recipients. Ongoing secondary longitudinal studies (the High School Longitudinal Study of 
2009 freshmen and the Education Longitudinal Study of 2002 sophomores) will incorporate 
questions on certification for out-of-school youth and young adults. 

Census. The Census Bureau has added a topical module on credential attainment 
based on the ATES Pilot Study to the 13 th wave of the 2008 panel of the Survey of Income 
and Program Participation (SIPP). The module was fielded in the fall of 2012 for release in 
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fall of 2013. These data will provide the first official statistics from the federal government on 
the prevalence of industry-recognized certifications and occupational licenses. The module 
is available at http://www.reqinfo.gov/public/do/PRAViewlC7ref nbr=201 202-0607- 
002&iclD=1821 15 . In addition, Census has identified an opportunity for further testing of 
certification/license items in the field test of the redesigned SIPP. The inclusion of these 
items in a different survey context will test the robustness of the items across platforms. 

Other GEMEnA member agencies, including BLS and NSF, are considering 
placement of certification/license items in their household studies related to education and 
training for work. GEMEnA will continue to identify opportunities to recommend the inclusion 
of validated survey items in appropriate federal household data collections. 

While survey data from US adults can tell us a great deal about the prevalence 
and key characteristics of industry-recognized certifications, administrative data from 
organizations that issue certifications or those that accredit the certification process could 
provide a richer information. During the ATES Pilot Study, GEMEnA worked with the two 
major US accrediting agencies, the American National Standards Institute (ANSI) and the 
Institute for Credentialing Excellence (ICE), to identify certifying bodies willing to share data 
for the seeded sample. 26 ANSI and ICE cannot be used, however, as a source of 
administrative data on certifications; in addition to legal issues concerning confidentiality, 
these organizations do not include the universe of certification providers. It is also infeasible 
at this time to collect administrative data from individual credentialing bodies. Nonetheless, 
the movement towards accreditation and standards could pave the way for future access to 
these records, yielding significant new information for research and policy related to 
certifications. 

4.3.2 Strand 2: Certificates 

GEMEnA’s second strand of work is to develop and deploy of a core set of survey 
items related to the prevalence and key characteristics of subbaccalaureate educational 
certificates. Due to the inconclusive results of the ATES Pilot Study data on certificates, 
GEMEnA has recommended additional developmental work on survey items related to 


26 ANSI ( http://www.ansi.org ) has developed a process to accredit credentialing systems that lead to “better educated 
and qualified workers.” ICE ( www.credentialinqexcellence.org ) develops standards for both certification and certificate 
programs and serves as a clearinghouse for information on trends in certification. 
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certificate attainment. In addition, NCES is responding to increased interest in 
subbaccalaureate educational certificates by improving the ability of its current data 
collections to measure these credentials. The three projects described below will improve 
and expand the ability of NCES studies to answer important policy questions about the 
prevalence and economic impact of certificate education. 

1. Developing a Valid Survey Item to Enumerate Certificates in Federal Surveys 

A new pilot study is being designed to improve on the measurement of 
subbaccalaureate certificates that was attempted in ATES. The work will include a sharper 
focus on measures specific to certificates and will feature a larger and cleaner sample of 
known certificate holders. The study will begin with focus groups to determine how 
certificate holders talk and think about their credential. Next, questionnaire items will be 
developed and tested in cognitive interviews. Finally, the new questions will be tested in a 
pilot study using a larger sample of known certificate holders that is more representative 
both geographically and by field of study than was the seeded sample in the ATES pilot 
study. Paired with a new national sample, this new seeded sample will provide a more 
rigorous test of the validity of the new certificate items. This work is expected to take 
approximately 18 months. 

2. Improving Institutional Data Collection on Certificates 

NCES currently collects annual counts of the number of certificates awarded, 
through its Integrated Postsecondary Education Data System (IPEDS). IPEDS gathers 
information from every college, university, and technical and vocational institution that 
participates in federal student aid programs — over 7,000 postsecondary institutions in all. 
The IPEDS data collection is guided by the National Postsecondary Education Cooperative 
(NPEC), which oversees IPEDS research and development. 

NPEC has recently created a working group to look at how institutions report 
certificate completions to IPEDS in order to improve the quality, comparability, and 
usefulness of these data. To support this work, an NPEC working group recently conducted 
a study of certificate reporting practices. The working group began meeting in October 
201 1 ; a final report was released in September 201 2 (Sykes 2012). 
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GEMEnA supports the recommendations of the working group to 1) clarify IPEDS 
instructions and definitions for certificates; 2) convene a Technical Review Panel to explore 
the feasibility of modifying and expanding IPEDS certificate categories; and 3) collect 
noncredit certificate data. Improvements in institutional reporting of certificate awards and 
the addition of data on noncredit certificates would greatly enhance the accuracy and utility 
of administrative data on subbaccalaureate educational credentials. 

3. Oversampling Certificate Holders in BPS: 12 

Since 1996, NCES has conducted a periodic longitudinal study of students who have 
recently matriculated into postsecondary education, the Beginning Postsecondary Students 
Study (BPS). The BPS is conducted about every eight years, with the most recent study 
initiated in 2012. BPS is the largest federal survey that regularly examines the economic 
outcomes of students in subbaccalaureate programs. However, due to sample size 
limitations, previous rounds of BPS have lacked the capacity to conduct in-depth analyses 
of the economic returns to educational certificates in specific fields of study. Starting in 
2012, BPS features a larger sample size for beginning students who enter postsecondary 
education with the goal of attaining an educational certificate. This oversample will allow 
finer-grained analysis of the characteristics of students in such programs, their persistence 
and attainment, and their occupational and economic outcomes. 

4.3.3 Strand 3: Enrollment and Participation 

GEMEnA’s third strand of work is to consider new and revised measures of 
participation in education and training designed to prepare out-of-school youth and adults 
for work. Existing federal data collections with survey items on enrollment and participation 
will benefit from a freshening of questionnaire items to better reflect current trends in 
delivery (including the expansion of on-line learning) and expanded policy interest related to 
federal and state investments in education and training for work. To begin this strand, 
GEMEnA has commissioned a background paper describing types of education and training 
for work that occur outside of traditional credit-bearing postsecondary education. The paper 
will focus on two high-frequency types of work-related education and training: noncredit 
coursework and formal on-the-job training. In consultation with experts, GEMEnA will then 
identify key research and policy questions and focus its development efforts — expected to 
parallel those for certifications/licenses and certificates — on high priority data needs. Along 
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with the first two stands of work, this strand will support the development of a new federal 
household survey focusing on education, training, and credentials (strand 4). 

4.3.4 Strand 4: Household Study on Education, Training, and Credentials for Work 

GEMEnA’s fourth strand of work is to support NCES in the development of a new 
household study on education, training, and credentials for work. This strand of work is 
proceeding in two stages. First, NCES will evaluate the feasibility of using a mail-out 
methodology to conduct a household survey on education, training, and credentials for 
work. Second, assuming a positive outcome on the first step, NCES will proceed with 
planning for a full-scale household data collection on US adults ages 16 to 65. 

Due to growing problems reaching survey respondents by telephone, NCES has 
recently shifted its household data collection program from a telephone-based collection to 
a mail-based collection. The mail-based methodology has been found to yield acceptable 
response rates for surveys asking about children’s education, but is untested for surveys 
asking adults about their own education and training. Thus, the first stage in this strand is to 
determine whether this methodology is feasible for such a survey. For this purpose, 
GEMEnA has developed the National Adult Training and Education Survey (NATES) Pilot 
Study survey instrument. The NATES Pilot Study will collect in-depth information on 
credentials related to work, including formal educational attainment, industry-recognized 
certifications, occupational licenses, and formal apprenticeships — based in part on the 
developmental work in strands 1 and 2. It also collects detailed information about the 
education and training that adults participate in to acquire the skills and knowledge needed 
for jobs based on the developmental work in strand 3. Finally, NATES Pilot Study includes a 
set of employment and demographic items derived in part from strands 1-3 work and in part 
from existing Census Bureau data collections. Because the question-development work in 
stands 1-3 has not been completed, the Pilot Study version of the NATES instrument 
includes many test and placeholder questions, but the overall instrument is designed to 
approximate the structure, content, and length desired for a final household survey on 
adults’ education, training, and credentials for work. 

This NATES instrument will be used in a pilot test of the mailout methodology to be 
conducted in the winter and spring of 2013, The 2013 NATES Pilot Study will examine 
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response rates at both the unit and item level and includes a nonresponse bias study 
based on in-person interviews with a sample of nonrespondents. The Census Bureau is 
conducting the data collection on behalf of NCES, collecting information from adults ages 
16 to 65 in approximately 10,000 households. 

If response rates on the NATES Pilot Test are sufficiently high or if nonresponse bias 
is deemed to be sufficiently negligible, and pending availability of funds, NCES could field a 
production NATES in the near future. The production version of NATES would be 
developed after completion of all work in strands 1-3, so that it could incorporate the 
lessons learned from all of GEMEnA’s work; it would provide the synthesis and culmination 
of these work strands, and would initiate a new federal data collection to meet pressing 
research and policy data needs. 

4.4 Ensuring Relevance to Policy and Research 

GEMEnA began its work with a clear mandate from the fall 2009 Brookings Institute 
meeting: develop new federal survey measures of industry-recognized certifications and 
subbaccalaureate educational certificates. As the member agencies now seek to deploy 
developed items to relevant federal surveys, continue developmental work, and begin a new 
focus on participation in education and training for work, GEMEnA wants to ensure the 
relevance of its efforts for answering critical policy and research questions. To this end, 
GEMEnA has established an Expert Panel to offer guidance and support on the four strands 
of work described in section 4.3. The Expert Panel is comprised of senior researchers and 
policymakers who use data to understand the role of education and training for work and 
meets annually beginning in the fall of 2012. The Expert Panel’s initial membership is 
listed below. 


GEMEnA Expert Panel Members 


Thomas Bailey 
Anthony Carnevale 
Evelyn Ganzglass 
Morris Kleiner 
Christopher Mullin 
James Parker 


Kenneth Poole 
Andrew Reamer 
Jesse Rothstein 
Jim Van Erden 
Holly Zanville 
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A. Details on the Interagency Working Group on 
Expanded Measures of Enrollment and Attainment 

(GEMEnA) 

Beginning in December 2009, the federal Interagency Working Group on Expanded 
Measures of Enrollment and Attainment (GEMEnA) convened monthly to develop new 
survey measures of certifications, licenses, and certificates. A timeline for the activities of 
the GEMEnA is provided in table A-1. The GEMEnA represents a broad range of federal 
agencies committed to improving the measurement of work-related credentials. Current 
members of GEMEnA, as well as past members from each participating agency include: 

Census Bureau 

Bob Kominski 
Stephanie Ewert 

Bureau of Labor Statistics 

Harley Frazis 

Dori Allard 

Tom Nardone (past) 

Office of Management and Budget 

Shelly Martinez 
Rachel Zinn 

Department of Education, Office of the Undersecretary 

Jon O’Bergh 

Council of Economic Advisors 

Chinhui Juhn 
Elizabeth Ananat (past) 

Sarena Goodman (past) 

Jesse Rothstein (past) 

Chuck Pierret(past) 
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National Science Foundation 

Dan Foley 
John Finamore 

National Center for Education Statistics 

Sharon Boivin 
Sarah Crissey 
Lisa Hudson 
Kashka Kubzdela 
Isaiah O’Rear 
Matthew Soldner 
Andrew Zukerberg 
Tom Weko (past) 


Table A-1 . Timeline for GEMEnA activities: 2009-1 1 


Activities 

Date 

Reviewed research and data available on credentials, and developed a 
bank of existing survey items 

October 2009 

Developed focus group protocols and conducted certificates and 
certifications/licenses focus groups 

November to December 2009 

Discussed focus group results and developed plans for cognitive 
interviews 

January to March 2010 

Began planning for pilot study; discussed pilot study parameters, 
mode, and key measures 

April to May 2010 

Developed draft pilot study questionnaire items 

April 2010 

Conducted cognitive interviews 

May 2010 

Discussed results of cognitive interviews and final pilot study 
questionnaire 

June 2010 

Conducted pilot study 

September 2010 to January 201 1 

Discussed pilot study analysis plans 

September 2010 

Discussed initial pilot study results 

February and March 2011 

Reviewed preliminary analysis tables 

May to June 2011 

Discussed item inclusion in other surveys and future work 

August 2011 

Reviewed final report and provided recommendations 

September to November 201 1 


SOURCE: GEMEnA Internal Records, 2009-11. 
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B. ATES Pilot Study Design and Methodology 

B.1 Introduction 

The Adult Training and Education Survey (ATES) Pilot Study is being used to 
evaluate the quality of data received using newly developed certification/license and 
certificate measures. It was not a goal of the Pilot Study to generate national estimates 
for publication. Rather, information gathered from it will ultimately be used to make 
recommendations for a core set of items that can be used by the National Center for 
Education Statistics (NCES) and other federal statistical agencies to measure the 
prevalence of professional certifications/licenses and education certificates in the 
United States. 

In the process of designing the ATES Pilot Study, the primary objective was a design 
that favored timeliness and sample yield over high response rates. In an effort to boost yield 
while limiting additional costs, a few methodological experiments were embedded in the 
design. These experiments also allowed for an assessment of the effect of various 
treatments on response rates. This appendix serves as documentation of the study’s design 
and operations and provides the results of the methodological experiments. Section B.2 
contains an overview of the study and sample design. The data collection methods are 
detailed in section B.3. Item response rates and imputation, and unit response rates and 
weighting, are discussed in sections B.4 and B.5, respectively. The methodological analysis 
is presented in section B.6. Section B.7 contains a discussion of the study results and 
presents considerations that might be relevant to the design of future studies. 

B.2 Study and Sample Design 

The ATES Pilot Study included two samples: an address-based sample (ABS) of 
18,750 addresses that represents the 50 states and the District of Columbia and a 
convenience sample (or “seeded sample”) of 1,250 individuals with known credentials. 1 
The purpose of the national sample was to provide a representative sample for testing 
ATES Pilot Study survey items that examine education issues that have not been 


1 As is discussed in section C.2.1 .2, a total of 800 additional cases corresponding to individuals with known 
credentials were loaded for telephone interview attempts. 
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adequately studied through other data collection efforts. The seeded sample was designed 
to evaluate the rate of underreporting on reports of certificates, certifications, and licenses 
and to examine the measurement properties of items related to the classification of these 
credentials. 

Households were sampled as described below, and a screener 2 survey was 
administered by mail to an adult household respondent. Following completion of the 
screener by mail, one eligible adult per household was selected, and an extended survey 
was conducted by telephone with the sampled adult. 

In order to minimize household response burden, only one adult per household was 
sampled for the extended survey. In households where there was more than one eligible 
adult, within-household sampling procedures were implemented to select only one of them. 
However, if there was more than one eligible adult in a household, the sampled adult was 
also designated to serve as a proxy for another adult in the household in order to evaluate 
potential measurement error in proxy responses; after completing the extended interview 
about himself or herself, the sampled adult was asked to provide information about the 
proxy subject’s participation in certificate/certification/license programs. (See section B.2.3 
below for further details on proxy response.) 

This section describes the sample design and selection for the ATES Pilot Study. 
Section B.2.1 discusses the sampling of households for the study, followed by a discussion 
of the procedural details for selecting both households and adults within households for the 
nationally representative and seeded samples (sections B.2.1 .1 and B.2.1. 2, respectively). 
Section B.2.2 examines the sample size requirements needed for study purposes. Section 
B.2.3 discusses proxy response, and section B.2.4 details the expected and actual sample 
yields. 

B.2.1 Sample Selection 

As noted above, the ATES Pilot Study consisted of two samples: a nationally 
representative sample of addresses and a seeded sample of individuals with known 


2 While the term “screener” is used here (to adhere to conventional terminology), it is a bit of a misnomer in this 
context, in that it is actually an enumeration instrument and not a screening device. 
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credentials. The selection of the nationally representative sample is discussed in section 
B.2.1.1, and the procedures used to identify seeded sample participants are described in 
section B.2.2.2. 

B.2.1.1 Nationally representative sample. The nationally representative sample of 
18,750 addresses was drawn in a single stage from a file of residential addresses 
maintained by a vendor, based on the U.S. Postal Service (USPS) Computerized Delivery 
Sequence (CDS) file. To accommodate the use of computer-assisted telephone 
interviewing (CATI) to administer the extended survey, the sample of addresses was 
reverse-matched to landline telephone directories; a telephone number was obtained 
through the reverse match for 63 percent of the sampled addresses. 

For this nationally representative portion of the ATES sample, in each household 
that had completed a screener, one adult was randomly selected from all eligible adults. To 
be eligible, an adult had to be age 18 or older. As completed screeners were returned, they 
were scanned for data capture. The sampling was done using a preloaded random number 
that was attached to each address. That random number — a pseudo-random draw from a 
uniform (0,1 ) distribution — was multiplied by the count of eligible adults to determine which 
adult to select. 

Sampling one adult per household had the advantage of minimizing intrahousehold 
burden. Additionally, with this approach, in contrast to an approach that selects more than 
one adult in some households, any intrahousehold correlations — either in certificate 
credentials or certification/license participation (or in measurement error in the responses to 
the certificate/certification/license questions) — do not adversely affect the precision of 
estimates. 

B.2.1 .2 Seeded sample. A convenience sample of approximately 1 ,250 
respondents with known credentials was targeted in order to evaluate the rate of 
underreporting on reports of certificates, certifications, and licenses and to examine the 
measurement properties of items related to the classification of these credentials. This 
seeded sample of 1 ,250 respondents was expected to be sufficient to detect an 
underreporting rate (i.e. , the proportion of persons with certificates/certifications/licenses 
who fail to report the certificates/certifications/licenses) as small as 2 percent. 
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The sampling frame for the seeded sample was developed from 27 files of 
certification and license holders from one state and five national accrediting bodies. It is not 
possible to identify the specific credentialing bodies for disclosure reasons. These files 
included the credential holder’s name and address and the name/subject of the credential. 

A telephone number was available from these source files for 74 percent of the seeded 
sample cases; this was used to accommodate the use of the telephone to administer the 
extended survey. 

The original set of files of certificate and certification/license holders contained a total 
of 33,495 records. After cleaning (to eliminate records with incomplete or invalid address 
information) and eliminating duplicate files (i.e., those with the same street address), a total 
of 23,101 records remained. Target sample sizes from each file were established with the 
goal of obtaining sufficient numbers of certificate and certification/license holders from a 
variety of fields. The seeded sample contains state license holders in alcohol- and drug- 
related fields; electrical, pharmaceutical, plumbing, real estate, and energy-related fields; 
certifications/licenses in lab technology, radiation, and other health-related fields; and 
community college certificates covering a variety of programs. 

In order to facilitate data collection procedures for the seeded sample that were 
comparable to those for the nationally representative sample, small modifications were 
made that aimed to ensure that the person in the seeded sample source file was the 
“sampled adult.” (For cases in which none of the enumerated household members 
appeared to match the name of the person listed in the seeded sample file, one adult was 
randomly selected from the set of eligible adults in the household and the uncertainty in the 
match was flagged in the database.) 

Due to lower-than-expected yield for the seeded sample, on January 12, 2011, a 
supplemental seeded sample (800 cases) was loaded into the CATI system in an effort to 
increase the number of sampled adult interviews completed with seeded sample cases. 
Unlike the original seeded sample, these supplemental cases were not mailed screeners. 
Instead, the adult listed in the seeded sample source file was loaded as the “sampled adult,” 
and the telephone number available from the seeded sample source file was used to 
contact the sampled adult for the extended survey. 
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B.2.2 Sample size requirements 

The key objective of the ATES Pilot Study was to assess the measurement 
properties of a new series of items about certifications/licenses and certificates. Thus, 
during the design of the ATES Pilot Study, it was determined that the sample requirements 
should enable the generation of cross-sectional estimates of certification/license and 
certificate holders in the population (although there were no plans to publish these 
estimates), allowing for analysis of a series of 10 to 15 items for some key subgroup 
analysis. 

Table B-1 shows the expected numbers of completed extended surveys for 
subgroups defined by credential status, educational attainment, age, and race/ethnicity. 
MacCallum et al. (1999) cite various recommendations in the literature for minimum sample 
sizes. Although these sample size recommendations vary, there appears to be a general 
consensus among the source papers cited that a sample size of 500 is generally adequate, 
and some authors cited indicate that sample sizes in the range of 100 to 250 may be 
sufficient. While the sample sizes shown in table B-1 are not sufficient to support analyses 
by race/ethnicity for adults with educational attainment of Associate’s degree, some college, 
or other postsecondary less than B.A., they are generally large enough to support analyses 
by credential status, by age, and for the educational attainment subgroups shown in the 
table. Additionally, analyses of some subgroups of adults with less than a bachelor’s degree 
are supported. Thus, the expected sample sizes shown in table B-1 were deemed sufficient 
for the desired analyses. 

B.2.3 Proxy response 

It is often difficult in household surveys to contact and interview a specific adult 
householder. For this reason, a response from another adult in the household is sometimes 
used as a proxy for the specific sampled adult or for all adults in the household. There are 
concerns about the ability of proxy responders to report accurately about certifications/ 
licenses and certificates. The ATES cognitive research suggested that spouses and 
partners feel confident in reporting some basic information by proxy (such as whether or not 
their spouse or partner has a certification/license and the occupational field), but not 
detailed information (such as credit hours or provider). 
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The ATES Pilot Study was used to empirically evaluate the feasibility of using a proxy 
responder to report on certifications/licenses and certificates. Conducting proxy interviews 
about adults facilitates the analysis of item-by-item differences between item response rates 
for self-reports versus for proxy reports, as well as the percentage of “don’t know” 
responses. 


Table B-1. Estimated percentage of adult population and expected number of 
completed ATES sampled adult surveys, by key subgroup 
characteristics: 2005 


Key subgroup characteristics 

Estimated percentage 
of adult population 

Expected number of 
completed ATES 
surveys 

Total 

100 

4,505 

Credential status 

Has certificate, certification, or license 1 

24 

1,092 

Educational attainment 

High school diploma or less 

66 

2,995 

Associate’s degree, some college, other less than B.A. 

6 

277 

Age 

18 to 30 years old 

23 

1,023 

31 to 45 years old 

29 

1,328 

46 years old and over 

48 

2,155 

Select characteristics by educational attainment 
Less than bachelor’s degree 

Has certificate, certification, or license 1 

14 

621 

Hispanic 

11 

481 

Black, Asian, American Indian, or Pacific Islander 

14 

629 

White 

48 

2,162 

Associate’s degree, some college, other less than B.A. 

Has certificate, certification, or license 1 

2 

105 

Hispanic 

1 

32 

Black, Asian, American Indian, or Pacific Islander 

1 

58 

White 

4 

188 


1 The percentage of adults who reported that their occupation has legal or professional requirements for continuing education or 
training. 

NOTE: Hispanic includes Latino, Black includes African American, American Indian includes Alaska Native, and Pacific Islander 
includes Native Hawaiian. Race categories exclude persons of Hispanic ethnicity. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Education for Work-Related Reasons 
Survey of the National Household Education Surveys Program (NHES), 2005. 
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Proxy respondents were identified using a two-pronged approach: 


• After the sampled adult completed the interview about himself or herself, he/she was 
asked to complete an interview about a preidentified proxy subject (who is another 
adult household member). 

• Once the calling protocol was fulfilled, if personal contact was made on the final 
attempt but an interview could not be completed with the sampled adult, an eligible 
proxy respondent (another adult household member) was asked to complete an 
interview about the sampled adult. 

Both the sampled adult and the proxy subject were identified during sample selection 
based on the adult household members enumerated in the screener; all eligible proxy 
respondents were also identified based on the enumeration information provided in the 
screener. The 800 supplemental seeded sample cases that were loaded into the CATI 
system late in the data collection period were not eligible for proxy interviews. Because a 
proxy response could potentially have a negative impact on data accuracy, the proxy 
interviews can be distinguished in the data file from the interviews conducted with the 
sampled adult. Table B-2 shows the numbers of completed proxy interviews required to 
support the detection of 2 through 10 percent point differences in item response rates; 
various levels of item response rates are shown in this table. 

B.2.4 Expected and actual sample yield 

Screeners were sent to each of the 20,000 sampled addresses (excluding the 
800 supplemental seeded sample cases identified late in the data collection period). An 
expected screener unit response rate of 55 percent was assumed, which would have 
yielded 9,900 completed screeners. However, a total of only 9,113 completed screeners 
were returned, 8 of which did not have an eligible adult. 
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Table B-2. Number of completed proxy interviews required to support detection of 
various levels of differences in item response rates, by item response 
rate for self reports and percent difference in item response rates: 
2010-11 


Item response 
rate for self 
reports 

Percentage point difference in 
item response rates (self report 

Minimum number of completed proxy interviews required to 
detect difference in item response rates 

vs. proxy report) 

Based on a one-tailed test 

Based on a two-tailed test 

80 percent 

2 percent 

1,528 


2,501 

80 percent 

3 percent 

596 


891 

80 percent 

4 percent 

328 


479 

80 percent 

5 percent 

211 


305 

80 percent 

6 percent 

149 


213 

80 percent 

7 percent 

111 


159 

80 percent 

8 percent 

87 


124 

80 percent 

9 percent 

70 


99 

80 percent 

10 percent 

57 


82 

85 percent 

2 percent 

1,181 


1,861 

85 percent 

3 percent 

485 


717 

85 percent 

4 percent 

273 


396 

85 percent 

5 percent 

179 


257 

85 percent 

6 percent 

127 


183 

85 percent 

7 percent 

96 


138 

85 percent 

8 percent 

76 


108 

85 percent 

9 percent 

62 


88 

85 percent 

10 percent 

51 


73 

90 percent 

2 percent 

826 


1,255 

90 percent 

3 percent 

362 


528 

90 percent 

4 percent 

211 


304 

90 percent 

5 percent 

141 


202 

90 percent 

6 percent 

103 


147 

90 percent 

7 percent 

79 


112 

90 percent 

8 percent 

63 


90 

90 percent 

9 percent 

52 


74 

90 percent 

10 percent 

44 


62 

95 percent 

2 percent 

474 


696 

95 percent 

3 percent 

229 


329 

95 percent 

4 percent 

141 


202 

95 percent 

5 percent 

99 


141 

95 percent 

6 percent 

74 


106 

95 percent 

7 percent 

59 


83 

95 percent 

8 percent 

48 


68 

95 percent 

9 percent 

40 


57 

95 percent 

10 percent 

35 


49 


NOTE: Differences are expected to be detectable at the level of a = .05. 


B-10 




It was expected that a telephone number would be available (either from the vendor 
match or from a response to the telephone number question in the screener) for 70 percent 
of households with a completed screener, or about 6,930 households. As a result, the 
extended survey was expected to be attempted with about 6,930 adults. In actuality, a 
telephone number was available (either from the vendor match or from the screener) for 
96 percent of sampled adults (8,774 sampled adults). Late in the data collection period, 
an additional 800 records with phone numbers were selected from the seeded sample 
sampling frame to increase the sample yield, for a total of 9,574 phone numbers fielded for 
the extended survey. (These additional 800 seeded sample cases were not mailed a 
screener; for these cases, data collection began with the extended interview.) 

For the extended survey, a unit response rate of 65 percent was assumed, which 
would have yielded about 4,505 completed interviews. However, only 4,073 respondents 
completed extended surveys; 3,730 were from the national sample and 343 were from the 
seeded sample. 

Table B-3 summarizes the number of screeners and extended surveys expected to 
be and actually completed in the ATES Pilot Study. 


Table B-3. Number of household screeners and extended surveys fielded in the 
ATES Pilot Study, by count of expected and actual number completed 
and survey type: 2010-11 


Survey type 


Number completed 

Number fielded 

Expected 

Actual 

Household screeners 

20,000 

9,900 

9,113 

Extended surveys 

9,574 

4,505 

4,073 

Sampled adult interviews completed with sampled adult 

— 

— 

2,823 

Sampled adult interviews completed with eligible proxy 

— 

— 

1,142 

Proxy subject interviews completed 

— 

— 

108 


— Not available. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 
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B.3 Data Collection Methods 


The ATES Pilot Study was conducted using two complementary survey systems. 
Hard copy screener questionnaires were designed and captured using TeleForm, software 
for document scanning and imaging. The sampled adult extended interviews were 
conducted using a CATI system. For the screener, the following features were important for 
the ATES Pilot Study: 

• Forms design: Questionnaires were created using the Teleform Designer module. 
Form templates were used to classify each data entry field, and completed hard 
copy forms were processed by TeleForm to capture the responses. 

• Receipt control: The case management system provided for automatic receipt 
control in a flexible manner that was used to produce status reports that allowed 
ongoing monitoring of the survey’s progress. 

• Image preprocessing: TeleForm applied image preprocessing to the forms to 
correct any skewing that may have occurred during scanning or faxing and to 
remove other unwanted marks from the forms. 

• Data capture: TeleForm read the form image files and extracted data according to 
rules established for each questionnaire. TeleForm can recognize handwritten, 
printed, check box, and “bubble” data types. 

• Verification: Extracted data were subjected to validation. If a data value violated 
validation rules, the data were interactively reviewed by verifiers, and validation 
errors were resolved. 

The most important features of the CATI system for the ATES Pilot Study were the 
following: 

• Scheduling: The CATI scheduler was used to route telephone numbers to 
interviewers, maintain a schedule of callback appointments, and reschedule 
unsuccessful contact attempts to an appropriate day and time. 
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• Skip patterns: The CATI system automatically guided interviewers through the skip 
patterns in the questionnaire, reducing the potential for interviewer error and 
shortening the questionnaire administration time. 

Section B.3.1 below contains details about the data collection procedures for the 
screener. The extended survey data collection protocol and procedures are described in 
section B.3.2. Section B.3.3 contains a discussion of data collection procedures aimed at 
attaining high response rates. Section B.3.4 discusses data editing procedures. 

B.3.1 Screener data collection procedures 

Table B-4 shows the schedule for the screener mailings, as well as the number of 
screeners sent in each mailing, by sample. An initial screener was mailed via First-Class 
mail to each sampled address on September 15, 2010. Each initial screener mailing 
contained a $2 cash incentive. One week after the initial screener mailing, a thank 
you/reminder postcard was mailed to each address. A follow-up screener was mailed via 
First-Class mail on October 6, 2010, to the 13,750 addresses that did not respond to the 
first screener mailing. A second follow-up screener was sent via FedEx 2-Day 3 on October 
20, 201 0, to the 1 1 ,393 addresses that did not respond to the initial or first follow-up 
screener mailings. CATI data collection for the extended interviews was conducted between 
October 20, 2010, and January 19, 2011. 

Table B-4. Numbers of screeners mailed in each wave of the ATES Pilot Study, by 
sample type: 2010-11 


Sample type 

Initial screener mailing 
(09/15/2010) 

First follow-up 
screener mailing 
(10/06/2010) 

Second follow-up 
screener mailing 
(10/20/2010) 

Total 

20,000 

13,570 

11,393 

National sample 

18,750 

12,711 

10,666 

Seeded sample 

1,250 

859 

727 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 


The ATES Pilot Study contained an experiment to evaluate the relative effectiveness 
of three levels of promised incentives together with the effects of having informed the 
respondent in the screener mailing of the potential incentive for completion of the extended 


3 USPS Priority Mail was used for addresses that could not be delivered via FedEx (e.g., P.O. Box addresses). 
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interview and having provided a first stage prepaid incentive. Households were randomly 
assigned to $0 (no incentive), $10, or $20 incentive treatment groups (approximately 
20 percent to the $0 group, 40 percent to the $10 group, and 40 percent to the $20 group); 
these incentives were to be issued if the sampled adult completed the extended interview. 
All adults assigned to the $10 or $20 group who completed the extended survey were sent 
a check for the incentive amount. 4 

In order to provide information about the effect of the level of incentive and the effect 
of notification, these two conditions were tested experimentally. In about 60 percent of the 
screener mailings (75 percent of the 80 percent of households assigned to the $10 or $20 
incentive groups), the letter enclosed in the mailing notified the household that if an adult in 
the household was selected for the extended telephone survey, the adult would be offered a 
specified amount to complete the survey. Half of these respondents were notified of a $10 
incentive, and half were notified of a $20 incentive. Respondents were reminded of the 
incentive at the start of the telephone interview. In the remaining 25 percent of the 
households assigned to the $10 or $20 group, the screener letter contained no mention of 
the incentive for completing the extended interview, and respondents were notified of the 
incentive amount only when they were contacted for the extended interview. 

Although promised incentives have been shown in random-digit-dial (RDD) surveys 
to be less effective than prepaid incentives (Berk et al. 1987; Church 1993), their relative 
effectiveness in two-phase surveys such as the ATES Pilot Study is unknown. Due to the 
fact that the household will have already received an incentive in the initial screener mailing, 
and a relationship with the household will have already been established, it is possible that 
the relative effectiveness of the promised incentive to a prepaid incentive may be different in 
this context. Some of the results of the incentive experiment can be found in the 
methodological analysis in section B.6 below. 

B.3.2 Extended survey data collection procedures 

As noted earlier, the ATES Pilot Study extended survey interviews were 
administered using CATI technology. The CATI system was programmed to automatically 
guide the interviewers through the complex skip patterns contained in the survey. This 


4 A total of 136 adults refused to provide a name for the check and, therefore, were not mailed a check. 
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reduced the potential for interviewer error and helped to minimize the time for administering 
the interviews. The CATI system’s scheduling feature allowed cases to be automatically 
fielded for appointments and callback attempts to complete interviews not completed on the 
first call. Data were entered directly into the CATI database, which also contained the call 
history of each case. 

In the screener, the respondent was asked to provide a telephone number for each 
enumerated adult. Taking into account both the vendor-matched phone numbers and the 
respondent-provided phone numbers, a telephone number was available for 96 percent of 
sampled adults. For a given person, if both a vendor-matched telephone number and a 
respondent-provided telephone number were available, the extended survey was attempted 
using the respondent-provided telephone number. 

Telephone interviewers for the ATES Pilot Study were Westat interviewers who were 
identified to work on the study based on their experience and availability. All 27 interviewers 
who worked on the study had prior experience in Westat studies, and 18 had previous 
experience in education-related surveys. ATES Pilot Study interviewer training included a 
self-paced review of the training manual and program, training in the CATI system, a 
trainer-led WebEx session, and a supervised role-play to practice interviewing. 

Call attempts for the ATES Pilot Study were limited to 9 a.m. to 9 p.m. Monday 
through Friday, 2 p.m. to 6 p.m. Saturday, and 2 p.m. to 9 p.m. Sunday in the respondent’s 
local time. The maximum number of call attempts to complete an extended survey was 49, 
but 99 percent of households received fewer than 36 call attempts and 90 percent of 
households received fewer than 30 call attempts. The overall mean number of call attempts 
was 10, and the mean number of call attempts to complete an extended survey was 5.5. 
The average time to complete the extended survey was 13.1 minutes. 

For the first 2 months of CATI data collection, no refusal conversion was attempted 
for households that initially refused to complete the survey. In early December, due to 
lower-than-expected response rates, refusal conversion was implemented. 5 Approximately 
one-third of the interviewers — those with the highest cooperation rates, willingness, and 


5 Although attaining high response rates and minimizing nonresponse bias were not primary goals of this study, the 
lower-than-expected response rates were affecting sample yield. 
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availability — worked on refusal conversion. Prior to beginning the conversion efforts, these 
interviewers attended further training to refresh their skills in using refusal conversion 
strategies. Refusal conversion was attempted only on nonhostile refusals. Second refusals 
were considered final. 

Telephone interviewer performance was monitored throughout the field period. One 
important purpose of monitoring was to ensure high-quality interviews by reinforcing good 
interviewing practices and by helping to build interviewing skills through coaching. Monitors 
evaluated interviewers on their telephone manner and relationship with respondents, 
specifically on their level of skill in reading the questions, listening to the comments and 
questions of respondents and providing accurate probes and replies, correctly recording the 
information, and gaining respondent cooperation. The total time spent monitoring 
interviewers was approximately 230 hours over the extent of the CATI data collection 
period. 


Table B-5 presents the number of screener and extended interviews completed 
during each week of data collection. Forty-five percent of the screener interviews (4,141) 
were completed by October 3. By November 14 (the end of week 7), 8,927 screeners, about 
98 percent of the total, had been completed. The extended telephone survey interviews 
began in the third week of data collection. 
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Table B-5. Number and percentage distribution of ATES Pilot Study screeners and 
extended interviews completed, by week: 2010-11 


Week 


Screeners completed 


Extended interviews completed 

Weekending 

Number 

Percent 

Number 

Percent 

Total 


9,113 

100.0 

4,075 

100.0 

1 

October 3, 2010 

4,141 

45.4 

— 

— 

2 

October 10, 2010 

1,220 

13.4 

— 

— 

3 

October 17, 2010 

656 

7.2 

— 

— 

4 

October 24, 2010 

977 

10.7 

184 

4.5 

5 

October 31, 2010 

1,238 

13.6 

375 

9.2 

6 

November 7, 2010 

551 

6.0 

431 

10.6 

7 

November 14, 2010 

144 

1.6 

461 

11.3 

8 

November 21, 2010 

88 

1.0 

454 

11.1 

9 

November 28, 2010 

25 

0.3 

199 

4.9 

10 

December5, 2010 

29 

0.3 

462 

11.3 

11 

December 12, 2010 

22 

0.2 

334 

8.2 

12 

December 19, 2010 

8 

0.1 

346 

8.5 

13 

December 26, 2010 

3 

# 

130 

3.2 

14 

January 2, 2011 

4 

# 

194 

4.8 

15 

January 9, 2011 

4 

# 

267 

6.6 

16 

January 16, 2011 

3 

# 

179 

4.4 

17 

January 23, 2011 

0 

# 

59 

1.4 


— Not available. 

# Rounds to zero. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 

B.3.3 Data collection procedures aimed at attaining high response rates 

Although attaining high response rates was not a primary goal of the ATES Pilot 
Study, the design incorporated a number of features that may be used in an effort to 
maximize response rates. The following is a discussion of those features. 

Total design method/respondent-friendly design. This approach combined the 
attributes of the least expensive and best methods available, beginning with the least labor- 
intensive mode and moving to modes requiring increasingly greater amounts of labor. While 
this placed an emphasis on the use of resources, these procedures created a respondent- 
friendly approach that used design attributes, a scheduled sequence of contacts, and 
survey mode to motivate and encourage survey participation. Surveys that take advantage 
of a respondent-friendly design have demonstrated increases in survey response (Dillman, 
Smyth, and Christian 2008; Dillman, Sinclair, and Clark 1993). 
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Engaging respondent interest and cooperation. The content of respondent letters 
and frequently asked questions (FAQs) was focused on communicating the legitimacy and 
importance of the study. Interviewer training focused on strategies for communicating the 
importance and legitimacy of the survey and gaining cooperation. 

Nonresponse follow-up. The data collection protocol included several stages of 
nonresponse follow-up. In addition to the number of contacts, changes in follow-up method 
(mail, FedEx) were designed to capture the attention of potential respondents. Refusal 
conversion was used during the CATI portion of data collection to try to increase response 
rates. 


Flexibility in scheduling interviews. Whenever possible, telephone interviewers 
attempted to complete the extended interview at the time of first contact with the household. 
In situations where a respondent was unavailable, a call appointment was entered into the 
CATI management system with notations on the best time to reach the respondent. 

Incentives. Incentives were used at both the screener and extended interview 
levels. A prepaid incentive of $2 was used for the screener, and a promised incentive of $10 
or $20 was offered to a random subsample of sampled adults for completing the extended 
survey. (See section B.3.1 above for further details on the incentive experiment.) 

B.3.4 Data editing 

The ATES data collection process culminated in the delivery of edited data files and 
associated documentation. Limited data editing (correcting interviewer, respondent, and 
processing errors) was performed during the TeleForm processing (i.e. , the scanning of the 
hard copy screener forms) and during administration of the CATI extended interview. This 
section provides details about the edits that were performed. 

Returned screener forms were reviewed as the mail was opened to identify any 
forms with unusual processing issues, such as illegible marks across the page or damage 
that may have occurred in the mail. Those cases were flagged for special handling and 
supervisor review. All screener forms with at least one question answered were scanned 
into the TeleForm database. 
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The extracted data were subjected to field validation according to project 
specifications. Data items identified by TeleForm for additional review went through a 
verification process. In the verification process, a staff member trained in data verification 
business rules for ATES (see attachment B-1 to this appendix) reviewed the data items and 
posted updates as needed. All phone numbers provided in the screener were verified for 
accuracy. For other screener items, if a data value violated validation rules, the data were 
flagged for review by verifiers who interactively reviewed the images and the corresponding 
extracted data, and resolved validation errors. 

Frequencies and cross-tabulations of screener variables were reviewed during post- 
data collection data editing to ensure that the screener variables were complete, accurate, 
and error-free. Any data issues encountered during processing were noted in a data 
decision log and reviewed. Global data decisions, such as how to handle “mark one” 
responses when two answers were given by the respondent, are included in 
attachment B-1 . 

During the development of the CATI specifications for the extended interview, a 
limited number of range edits were included. Since a primary objective of the ATES Pilot 
Study was to evaluate data quality, there were very few edits to the CATI data other than 
those built into the survey instrument. The only data edits occurred as a result of corrections 
to minor CATI programming errors and affected very few cases. 

B.4. Item Response Rates and Imputation 

In the ATES Pilot Study, as in most surveys, the responses to some data items were 
not obtained for all interviews. There are numerous reasons for item nonresponse. Some 
respondents do not know the answer for the item or do not wish to respond for other 
reasons. Some item nonresponse arises when an interview is interrupted and not continued 
later, leaving items at the end of the interview blank. Item nonresponse may also be 
encountered because responses provided by the respondent are not internally consistent, 
and this inconsistency is not discovered until after the interview is completed; for ATES, 
any such inconsistencies remained in the data files since no consistency edits were 
administered. 
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For the ATES Pilot Study, imputation was used on a very limited basis to address 
item nonresponse. Data items needed for weighting, the key outcome variables, and select 
demographic variables were imputed. This imputation was done for sampled adult self- 
reported interview variables only; missing responses for proxy subject variables were not 
imputed. The final data file contains both the imputed and unimputed versions of each item 
that was imputed. The following variables were imputed: 

Screener 

SEX: Gender of the sampled adult 

Extended interview 

IN5: Worked at a job for pay or income in last 12 months 
CN1: Has professional certification or state/industry license 
EDI: Currently attending/enrolled in school 
ED3: Highest degree/level of school completed 
CT 1 : Has certificate 
AL1: Year born 

AL3: Spanish/Hispanic/Latino origin 
AL4: Race 6 

AL8: State/country/territory born 

AM12A: Household income range: $25K or less, > $25K 
AM12B: Household income range: $50K or less, > $50K 
AMI 21: Detailed household income range: $0-$25K 
AMI 22: Detailed household income range: $25K-$50K 
AM123: Detailed household income range: > $50K 
AM3: Own home/rent/other arrangement 

B.4.1 Item response rates 

For most of the imputed data items collected in the ATES Pilot Study, the item 
response rate was very high. Table B-6 shows the item response rates for the imputed 
items as well as the number of cases for which each item was attempted. The only items 


6 Separate binary variables were derived from AL4 that indicate the race or races reported. 
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with item response rates less than 90 percent are those pertaining to household income, 
which was expected due to the sensitive nature of the question. 

B.4.2 Imputation 

For the ATES Pilot Study, imputation was done for two reasons. First, complete 
responses were needed for the variables used in developing the sampling weights. Second, 
data users will be employing a variety of methods of analysis, and complete responses may 
aid some analyses. Separate preimputation and post-imputation versions of each imputed 
variable were provided in order to enable the data users to identify the imputed values. 
Imputation was performed only on the self-reported variables; proxy subject variables were 
not imputed. 

For several of the variables that were imputed, a hot-deck procedure was used to 
impute missing responses. In this approach, the entire file was sorted into cells defined 
by characteristics that were likely to be associated with key outcome variables or with 
differences in item response propensities. Within each cell, cases were divided into 
two classes, depending on whether or not the item was missing. For an observation with a 
missing value, a value from a randomly selected donor (observation in the same cell, but 
with the item completed) was used to replace the missing value. This method is called a 
hot-deck procedure, because actual values are imputed from donors selected from the 
current dataset as opposed to an external dataset. 
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Table B-6. Potential responses and response rates for imputed items in the ATES 
Pilot Study: 2010-11 


Variable 

Label 

Potential responses 

Item response rate 
(percent) 

EDI 

Currently attending/enrolled in school 

4,073 

99.93 

IN5 

Worked at a job for pay or income in last 12 months 

4,073 

99.90 

CN1 

Has professional certification or state/industry license 

4,073 

99.48 

ED3 

Highest degree/level of school completed 

4,073 

99.48 

CT1 

Has certificate 

4,073 

99.46 

AL8 

State/country/territory born 

4,073 

98.87 

AL3 

Spanish/Hispanic/Latino origin 

4,073 

98.85 

AL1 

Year born 

4,073 

98.38 

AGE 

Respondent’s age 

4,073 

97.77 

SEX 

Respondent's gender 

4,073 

97.30 

AM3 

Own home/rent/other arrangement 

4,073 

96.88 

AL4C1 

Race White 

4,073 

96.76 

AL4C2 

Race Black or African American 

4,073 

96.76 

AL4C3 

Race American Indian or Alaska Native 

4,073 

96.76 

AL4C4 

Race Asian 

4,073 

96.76 

AL4C5 

Race Native Hawaiian or Other Pacific Islander 

4,073 

96.76 

AM 123 

Detailed household income range: >$50K 

1,961 

78.63 

AM121 

Detailed household income range: $0-$25K 

1,038 

74.37 

AM12B 

Household income range: $50K or less, > $50K 

2,627 

71.72 

AM 122 

Detailed household income range: $25K-$50K 

1,074 

71.69 

AM12A 

Household income range: $25K or less, > $25K 

3,407 

71.68 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 
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The characteristics used to form the hot-deck cells are referred to as sort variables 
or boundary variables. Two types of boundary variables were used. Hard boundary 
variables were considered to be so important that the donor and the recipient were required 
to match exactly. For the other boundary variables, called soft boundary variables, the 
values did not have to match exactly. The WesDeck software was used to implement the 
hot-deck imputation procedure. WesDeck is a proprietary SAS macro developed by Westat 
to form hot-deck cells, impute using the hot-deck method, and generate output to verify the 
imputation. 

Gender, year born, current enrollment status, highest degree/level of school, and 
race/ethnicity were imputed manually; all other items were imputed using the hot-deck 
procedure. Variables that are manually imputed typically involve complex relationships 
and/or constraints (for example, using information about household members who were not 
sampled) that would require extensive programming in order to impute using a hot-deck 
procedure. Furthermore, the reasonableness of imputed values for these characteristics can 
often be assessed by examining the values of these variables for other members of the 
household. The use of the manual imputation approach in this situation permits the review 
of the characteristics of household members when imputing the missing values. 

For manual imputation, three sort variables were used. State was used as the first 
sort variable; that is, whenever possible, all values were imputed from within-state donors. 
Because there is some geographic clustering of subpopulations within states, zip code was 
used as the second sort variable. Cases were then sorted by the household identification 
number. The general approach used to find a donor was to search upward in the sorted list. 
If no donor was found (within zip code, within state), a downward search was used. If there 
was no eligible donor within the same zip code, the zip code restriction was lifted and the 
search was expanded (first upward, then downward when necessary) within a state. 

For gender, either male or female was imputed with equal probability based on a 
random number. The variable AL1 (year born) was imputed using the adult’s age as 
reported in the screener. When the age from the screener was not available, the manual 
procedure described above was used to impute year born. For state/country/territory where 
the respondent was born, the hard boundary variable was the adult’s state of residence. 
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The hard boundary variables for the questionnaire items on certificate attainment 
and certifications and licenses (CT1 and CN1) as well as for home tenure (AM3) and all of 
the income variables were as follows: 

• AGECAT — a categorical variable derived from adult’s year born, with the categories 
18 through 29 years, 30 through 49 years, and 50 or older; 

• EDUC — a variable that indicates whether or not the adult has at least a high school 
diploma or the equivalent (derived from ED3); 

• EMPSTAT — a variable that indicates whether or not the respondent worked at a job 
for pay or income in the last 12 months (IN5). In deriving EMPSTAT, adults with 
missing values for IN5 were grouped together with adults reported not to have 
worked in the last 12 months. 

After values were imputed for all observations with missing values, the distribution of 
the items prior to imputation (i.e. , the respondents’ distribution) was compared to the post- 
imputation distributions of the imputed values alone and of the imputed values together with 
the observed values. 

For each data item for which any values were imputed, an imputed version of the 
variable was created that had the original variable name with “_l” attached to the end; for 
example, the original variable IN5 has a corresponding imputed variable IN5J. Both the 
original variables and the imputed variables are provided in the data files. This gives data 
users the ability to employ alternative imputation procedures or account for the imputation in 
computations of the reliability of the estimates produced from the dataset (see, e.g., Rao 
and Shao 1992 or Rubin 1987). For example, some users might wish to analyze the data 
with the missing values rather than the imputed values, in which case the original variables 
can be used. Additionally, missing values in the original variables could be replaced with 
other values imputed by some user-defined imputation approach. 

B.5. Unit Response Rates and Weighting 

Unit nonresponse is generally regarded as one measure of survey quality. Response 
rates are of concern because survey estimates could potentially suffer from nonresponse 
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bias if those who respond to a survey are very different from those who do not, and the risk 
of nonresponse bias is considered greater when unit nonresponse is higher. Although 
producing population estimates was not an objective of the ATES Pilot Study, unit response 
rates are presented here for the benefit of the data file user who is interested in these 
measures. 

There are no plans to release survey estimates from the ATES Pilot Study; the aim 
of the Pilot Study is to provide a large-scale methodological evaluation of survey measures. 
However, to facilitate computation of preliminary estimates of the number of certification/ 
license and certificate holders (overall and for key subgroups) so as to aid in the evaluation 
of the accuracy of the survey measures, weights were computed to account for differential 
within-household selection probabilities and to adjust for biases due to differential 
nonresponse to the screener or the extended survey and due to differential coverage. 

This section gives unit response rates and discusses weighting procedures for the 
ATES Pilot Study. Section B.5.1 defines unit response rates, followed by details of the 
screener (section B.5.1.1) and extended survey (section B.5.1. 2) unit response rates. 
Weighting procedures are discussed in section B.5.2. Section B.5.3 gives details on 
methods for computing sampling errors, including using replication (section B.5.3. 1) and 
Taylor series approximation (section B.5.3. 2). 

B.5. 1 Definition of unit response rates 

A unit response rate is the ratio of the number of units responding to the survey 
(for example, the units could be households or persons) to the number of units sampled and 
eligible for the survey. In some cases, these rates are easily defined and computed, while in 
other cases the numerator or denominator of the ratio must be estimated. 

For reporting the results from the ATES Pilot Study, the overall unit response rate 
indicates the percentage of possible surveys that were completed taking all survey stages 
into account, while the unit response rate measures the percentage of surveys that were 
completed for a specific stage of the survey. Specifically, household members were 
identified for interviews in a two-stage process. As described in section B.3.1, screener 
questionnaires were used to enumerate household members; as completed screeners were 
returned, one eligible adult was sampled in each household, and then the extended 
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interview was attempted (by phone) with the sampled adult. If a household member failed to 
complete the first stage screener, no member of that household could be sampled for the 
extended interview. Under this design, the unit response rate for the second stage 
(extended survey) is the percentage of sampled persons who completed these interviews. 
The overall unit response rate is the product of the first- and second-stage unit response 
rates (i.e., the screener unit response rate multiplied by the extended survey unit 
response rate). 

B.5.1.1 Screener response rates. As described earlier, screeners were sent to 
each of the 20,000 sampled addresses. The overall screener unit response rate was 
calculated as the ratio of the number of completed screeners to the total number sampled, 
excluding the cases with nondeliverable screener forms. (Nondeliverable addresses are 
treated as ineligible.) The rates given here are unweighted; however, since the sample 
of addresses selected into the national sample was an equal probability sample, the 
weighted and unweighted screener unit response rates for the national sample are identical. 
Table B-7 shows the distribution of sampled and completed screeners, by sample type. The 
overall screener unit response rate was 52 percent, slightly lower than the assumed 
55 percent screener unit response rate. There were a total of 9,1 13 completed screeners, 

8 of which did not have an eligible adult. For the national sample, a total of 8,478 screeners 
were completed, for a screener unit response rate of 51 percent; 635 screeners were 
completed among seeded sample cases, for a screener unit response rate of 55 percent. 


Table B-7. Number of sampled addresses and completed screeners, and screener 
unit response rates, in the ATES Pilot Study, by sample type: 2010-11 




Number completed 


Screener unit 

Sample type 

Number of sampled 
addresses 

Expected 

Actual 

response 
rate (%) 

Total 

20,000 

9,900 

9,113 

51.5 

National 

18,750 

— 

8,478 

51.2 

Seeded 1 

1,250 

— 

635 

55.3 


—Not available. 

1 Counts exclude the supplemental sample of 800 additional seeded sample cases (see section B.2. 1 .2). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 

Table B-8 shows the distribution of screener cases by type of return for cases that 
did not complete the screener. About 12 percent (2,301 screeners) of screener cases were 
classified as ineligible due to nondeliverable screener forms. About 41 percent of cases 
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failed to return a screener, less than 1 percent of cases returned blank screeners, and less 
than 1 percent returned screeners with annotations indicating refusal. 


Table B-8. Number and percentage of cases failing to return a completed screener, 
by sample type and type of return: 2010-11 


Sample type 

Type of return 

Number of cases 

Percent of cases 

Total 


10,887 

100.0 

All 

Nondeliverables (ineligible) 

2,301 

11.5 


Nonrespondents 




Form not returned 

8,257 

41.3 


Blank form returned 

173 

# 


Refused on form 

156 

# 

National 

Nondeliverables (ineligible) 

2,198 

11.7 


Nonrespondents 




Form not returned 

7,765 

41.4 


Blank form returned 

159 

# 


Refused on form 

150 

# 

Seeded 

Nondeliverables (ineligible) 

103 

8.2 


Nonrespondents 




Form not returned 

492 

39.4 


Blank form returned 

14 

1.1 


Refused on form 

6 

# 


# Rounds to zero. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 


B.5.1.2 Extended interview response rate. The extended interview unit response 
rate was calculated, for the national sample only, as the ratio of the number of completed 
extended interviews to the number of eligible sampled adults. For the national sample, a 
total of 3,730 extended interviews were completed, for unweighted and weighted extended 
interview unit response rates of 44 percent and 42 percent, respectively. (The expected 
extended interview unit response rate was 65 percent; see section B.7 below for discussion 
of factors likely to have affected this unit response rate.) Additionally, there were 343 
completed extended interviews from the seeded sample (including those completed among 
the 800 supplemental seeded sample cases that were released late in the data collection 
period). Because there are no reliable estimates of the size of the certifications/licenses and 
certificates population and the sampling frames used for the seeded sample were very 
selective (with, undoubtedly, very low coverage of this population), response rates were not 
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calculated for the seeded sample. Table B-9 shows the distribution of sampled and 
completed extended surveys, by sample type. 


Table B-9. Number of eligible adults sampled for the extended interview and 

number of completed extended interviews in the ATES Pilot Study, by 
sample type: 2010-11 



Number of eligible adults 

Number of completed extended interviews 


Sample type 

sampled for the extended 
interview 

Expected 

Actual 

Total 

9,905 

4,505 

4,073 

National 

8,471 

4,205 

3,730 

Seeded 

1,434 

300 

343 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 


Of the total 3,730 completed extended interviews from the national sample, 3,637 
were completed with the sampled adult himself or herself. Ninety-three of the interviews 
were completed about the sampled adult by an eligible proxy. Of the 3,637 interviews 
completed by the sampled adult about himself or herself, another 1,045 interviews were 
completed by the same sampled adult about a second eligible adult in the household. 

As mentioned, the focus of ATES, particularly for the seeded sample, was on sample 
yield and timeliness as opposed to response rates. As a result, techniques that could have 
been used to improve response were not employed in the extended survey administration. 
For example, no Spanish language surveys were administered, and refusal conversion did 
not begin until late in the extended interview data collection period. 

Table B-10 shows the distribution of reasons for nonresponse to the extended 
interview in the national sample. About 7 percent of extended interview nonresponse was 
due to the lack of an available phone number for the sampled adult, and 13 percent was 
because the phone number available was a “bad” match (a nonworking or nonresidential 
number or a number that was not associated with the sampled adult). About 32 percent 
(1,520 cases) of extended interview nonresponse was due to reaching the maximum 
number of call attempts after making contact with the household, about 19 percent were 
final noncontact cases, about 24 percent were refusals, and about 6 percent finalized with 
other reasons for nonresponse. 
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Table B-10. Distribution of reasons for nonresponse to the extended interview in the 
ATES Pilot Study (national sample only): 2010-11 


Reason for nonresponse 

Number 

Percent 

Total 

4,746 

100.0 

No phone number available 

315 

6.6 

Maximum calls 

1,520 

32.0 

Noncontact 

890 

18.8 

Nonworking, nonresidential, or mismatched phone number 

615 

13.0 

Refusal 

1,122 

23.6 

Other nonresponse 1 

284 

6.0 


1 Other nonresponse includes reasons such as English language or cognitive ability problems. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 

B.5.2 Weighting 

Weights were calculated for sampled adults in the ATES national sample only; 
weights were not provided for the seeded sample since this sample was purposively 
selected and was not intended to be used for inference to any particular subpopulation. 

For ATES, weighting was necessary to account for differential probabilities of selection for 
persons within households and to reduce potential bias due to differential nonresponse and 
differential coverage of subpopulations. 

For ATES, an eligible adult was defined to be a household member 18 years of age 
or older. Person-level weights were created for the sampled adult. The first step in 
computing the person-level weight was to compute the address-level base weight, which is 
the reciprocal of the probability of selection of the address. Since an equal probability 
sample of addresses was used for the national sample, the address-level base weight is a 
constant; 7 this weight is denoted as H. The person-level weight for sampled person j in 
household /', PW jt is the product of the address-level base weight and two weight 
adjustment factors: 

• A weight associated with sampling the person from among all eligible persons in the 
household, A . ; and 


7 For a very small proportion of households (roughly estimated from another study conducted by Westat to be about 
4 percent of households [Norman and Sigman 2009]), the household may have been selected at any of two or more 
addresses; for ATES, there was no attempt to measure the number of distinct addresses at which a household could 
have been selected. 


B-29 






• An adjustment associated with raking the person-level weights to American 

Community Survey (ACS) estimates of the number of persons in the target 

population, B.^ (see discussion of raking below). 

The procedures for computing the person-level weight adjustments are as follows. 

The first adjustment, which accounts for the probability of sampling person j from 
among all eligible adults in household i , is 

A ,=*, 

where N i is the number of eligible adults in household i . For each sampled person 

j , the unadjusted person-level weight, UPW . , can be written as the product of the 

household-level weight and the adjustment for within-household sampling. That is, for 
sampled person j in household i , the unadjusted person-level weight is 

UPWj = H -Aj. 

The second adjustment involves raking the person-level weights, SPW, to national 
control totals. Raking was proposed by Deming and Stephan (1940) as a way to ensure 
consistency between complete counts and sample data from the 1940 U.S. Census of 
population. For ATES, raking was used to correct for the bias due to nonresponse or 
noncoverage. The raking procedure was carried out in a sequence of adjustments: first, the 
base weights were adjusted to one marginal distribution (or dimension) and then the second 
marginal distribution, and so on. One sequence of adjustments to the marginal distributions 
is known as a cycle or iteration. The procedure was repeated until convergence of weighted 
totals to all sets of marginal distributions was achieved. (See Deming and Stephan [1940] 
for further details on raking and the convergence process.) For ATES, the raking iterations 
continued until the estimated totals were within 1 of all the control totals. 

Four dimensions were used for raking the ATES person-level weights: (1) a cross of 
number of persons in the household (exactly one/more than one) with age (18-29 years/ 
30-49 years/50 years or more) and sex; (2) a cross of number of persons in the household 
with the adult’s race/ethnicity (Black, non-Hispanic/Hispanic/other); (3) a cross of number of 
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persons in the household with highest educational attainment (less than high school 
diploma/high school diploma or the equivalent/some college); and (4) a cross of number of 
persons in the household with Census region (Northeast/South/Midwest/West). These 
raking dimensions were used because they included important analysis variables (e.g., 
educational attainment) and characteristics that are typically associated with nonresponse 
(e.g., region). The control totals used in this raking adjustment, shown in table B-11, were 
obtained from the 5-year 2005-09 ACS. 

The final person-level weight for each sampled person j is 


PW m =SPW r B m , 


where B . {k . is the raking adjustment factor for raking cell k and person j has the attributes 
corresponding to the levels of the dimensions of raking cell k . 


B.5.3 Methods for computing sampling errors 

In surveys with complex sample designs, such as the ATES Pilot Study, direct 
estimates of the sampling errors assuming a simple random sample will typically 
underestimate the variability in the estimates (Wolter 1985). The ATES sample design 
and estimation included procedures that deviate from the assumption of simple random 
sampling, such as sampling persons within households with differential probabilities and 
raking to control totals. 

B.5.3.1 Replication sampling errors. One method for computing sampling errors 
to reflect these aspects of the sample design and estimation is the replication method. 
Replication involves splitting the entire sample into a set of groups, or replicates, based on 
the actual sample design of the survey. The survey estimates can then be computed for 
each of the replicates by creating replicate weights that mimic the actual sample design and 
estimation procedures used in the full sample. The variation in the estimates computed from 
the replicate weights can then be used to estimate the sampling errors of the estimates from 
the full sample. 

A total of 80 replicates were defined for the ATES national sample, based on the 
sampling of addresses. This number was chosen to provide reliable estimates of sampling 
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errors with reasonable data processing time and effort. The specific replication procedure 
used was a jackknife replication method (Wolter 1985). It involved dividing the sample into 
80 random subsamples (replicates) for the computation of the replicate weights. The 
80 replicates were formed on the sampling order of the addresses. For each replicate, a 
replicate weight was developed using the same weighting procedures that were used to 
develop the full sample weight. 
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Table B-11. Control totals for raking the ATES person-level weights, by 
characteristics used in raking: 2010-11 


Characteristics 

Control totals 1 

Total 

219,423,725 

Number of persons in household, by age and sex 


One person in household 


18 to 29 years old 


Male 

2,036,107 

Female 

2,899,421 

30 to 49 years old 


Male 

5,850,267 

Female 

7,079,923 

50 years old and over 


Male 

6,820,442 

Female 

12,923,845 

More than one person in household, by age and sex 


18 to 29 years old 


Male 

21,565,302 

Female 

20,126,967 

30 to 49 years old 


Male 

35,499,210 

Female 

35,312,422 

50 years old and over 


Male 

34,185,192 

Female 

35,124,627 

Number of persons in household, by race/ethnicity 


One person in household 


Black 

6,249,931 

Hispanic 

3,024,686 

Other 

28,335,388 

More than one person in household 


Black 

18,853,684 

Hispanic 

26,002,526 

Other 

136,957,510 

Number of persons in household by educational attainment 


One person in household 


Less than high school diploma 

5,671,998 

High school diploma or the equivalent 

19,170,215 

Some college 

12,767,792 

More than one person in household 


Less than high school diploma 

27,812,662 

High school diploma or the equivalent 

94,450,782 

Some college 

59,550,276 


See notes at end of table. 
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Table B-11. Control totals for raking the ATES person-level weights, by 
characteristics used in raking: 2010-11 — Continued 


Characteristics 

Control totals 1 

Number of persons in household, by Census region 2 
One person in household 

Northeast 

7,013,500 

South 

8,855,790 

Midwest 

14,049,680 

West 

7,691,035 

More than one person in household 

Northeast 

33,542,149 

South 

39,464,681 

Midwest 

65,960,587 

West 

42,846,303 


1 The control totals are numbers of adults. 

2 The following states and the District of Columbia are in each Census region: Northeast: CT, MA, ME, NH, NJ, NY, PA, Rl, VT; 
South: AL, AR, DC, DE, FL, GA, KY, LA, MD, MS, NC, OK, SC, TN, TX, VA, WV; Midwest: IA, IL, IN, KS, Ml, MN, MO, ND, NE, 
OH, SD, Wl; West: AK, AZ, CA, CO, HI, ID, MT, NV, NM, OR, UT, WA, WY. 

NOTE: Black includes African American, Hispanic includes Latino, and Other includes all non-Black, non-Hispanics. Race 
categories exclude persons of Hispanic ethnicity. 

SOURCE: U.S. Department of Commerce, Bureau of the Census, American Community Survey (ACS), 2005-09. 

The jackknife variance estimator has the form 

u k = 1 

Where 6 is the population parameter of interest; 0 is the estimate of Q based on the 
full sample; 0 {k) is the estimate of Q based on the observations included in the kth replicate; 

and G is the total number of replicates. (For ATES, G = 80.) For ATES, the replicate 
weights are included in the file as FAWT1 through FAWT80. 

The computation of the sampling errors using these replicate weights can be done 
easily using the Windows-based software packages WesVar, SUDAAN (Shah et al. 1995), 
Stata, or AM Statistical Software; in WesVar, SUDAAN, or AM, the replication method 
should be specified as JK1. The current version of WesVar (version 5) is available from 
Westat; information can be obtained at www.westat.com . Information on obtaining SUDAAN 
can be found at www.rti.org/sudaan , and the AM software is available at am.air.org . 
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B.5.3.2 Taylor series approximation. Another valid approach to the estimation of 
sampling errors for complex sample designs is to use a Taylor series approximation. To 
produce standard errors using a Taylor series program, such as SUDAAN (Shah et al. 

1995), AM, or the survey data analysis procedures (e.g., PROC SURVEYMEANS, PROC 
SURVEYREG) in SAS version 9, two variables are required to identify the stratum and the 
primary sampling unit (PSU). The stratum-level variable is the indicator of the variance 
estimation stratum from which the unit (address or sampled person) was selected. The PSU 
is an arbitrary numeric identification number for the unit within the stratum. For ATES, the 
stratum variable is a dummy variable set to 1 for all records; the PSU variable was assigned 
sequentially so that each sampled address has a unique value. 

The PSU and stratum variables appear in the file as the variables TS_PSU and 
TSSTRAT. These variables can be used in SUDAAN to produce standard errors by 
specifying that the design is a “with replacement” sample (DESIGN = WR) and that the 
sampling levels are given by the appropriate stratum and PSU variables. For example, use 
TS STRAT and TS_PSU in the NEST statement. (Information on obtaining SUDAAN can 
be found at www.rti.org/sudaan .) In the SAS version 9 or higher survey procedures, the 
stratum and PSU variables are specified in the STRATA and CLUSTER statements, 
respectively. (Information on obtaining SAS version 9 or higher can be found at 
www.sas.com .) 

Stata also uses the PSU and stratum variables to define the units needed for 
computation of Taylor series variance estimates. (Information on obtaining Stata is available 
at www.stata.com .) To specify the stratum, PSU, and weight variables in Stata, use the 
svyset strata, svyset psu, and svyset pweight commands. For example, use the following 
commands to specify these design parameters: 

svyset strata TS_STRAT 
svyset psu TS_PSU 
svyset pweight FAWTO 

Data users should be aware that the use of different approaches or software 
packages in the calculation of standard errors may result in slightly different standard errors. 
Estimates of standard errors computed using the replication method and the Taylor series 
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method are nearly always very similar, but not identical. For a discussion of this issue, see 
Broene and Rust (2000). 

B.6 Methodological Analysis 

In order to determine the effectiveness of the design and methodology used for the 
ATES Pilot Study, several aspects of the survey were examined. These included an 
analysis of screener mail response rates after each wave of screener mailings (initial, first 
follow-up, and second follow-up); an analysis of several characteristics of screener 
respondents after each mailing stage; an examination and comparison of the effectiveness 
of the monetary incentive and whether or not the respondent was notified about the 
incentive on screener response, whether or not the respondent provided a telephone 
number, and extended interview response; an examination of characteristics of sampled 
adults by the source(s) of the telephone number(s); and an examination of screener 
response rates and phone number availability by type of address. 

Both weighted and unweighted results are presented in the tables in this section. 
Results pertaining to screener response and some screener characteristics contain 
unweighted estimates, while results pertaining to the extended interview contain weighted 
estimates. Tables displaying the types of telephones (landline vs. cell/wireless) associated 
with the sampled addresses also show weighted estimates. Chi-square statistics are 
presented where applicable, and all findings described in the text are statistically significant 
at the .05 level. 

B.6. 1 Screener response rates and characteristics of screener respondents 

Table B-12 gives the number of respondents and associated response rates after 
the initial screener mailing, the first follow-up screener mailing, and the second follow-up 
screener mailing, for both the national and seeded samples. The final overall screener 
response rate of 51 percent increased from 33 percent after the initial mailing and from 
42 percent after the first follow-up mailing. 
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Table B-12. Number of respondents and response rates for each wave and screener, 
by sample type: 201 0-1 1 



Initial screener mailing 
(wave 1 respondents only) 

First follow-up screener mailing 
(wave 1 and 2 respondents) 

Second follow-up screener 
mailing 

(wave 1, 2, and 3 respondents) 

Sample 

type 

Number 

Response rate 

Number 

Response 

rate 

Number 

Response 

rate 

Total 

6,055 

33.1 

7,476 

41.8 

9,113 

51.5 

National 


33.0 

6,969 

41.7 

8,478 

51.2 

Seeded 

415 

34.4 

507 

43.0 

635 

55.4 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 


While the screener follow-up mailings were effective in boosting the screener unit 
response rate, an important question is whether there is any indication that they reduced 
nonresponse bias. To examine this, various characteristics of the sampled adults obtained 
after each wave of screener mailings were tabulated; these are given in table B-13. The 
sampled adults resulting from the screener follow-up mailings differ from those in the initial 
screener in terms of age (those sampled as a result of the follow-ups were younger), 
educational attainment (the sampled adults from the follow-ups had lower levels of 
educational attainment), and type of phone number provided (those sampled from the 
follow-ups provided cell/wireless numbers in greater proportions). The Chi-square tests 
were conducted on the noncumulative samples. 

B.6.2 Incentive experiment results 

As discussed in section B.3.1, an experiment was conducted to assess the 
effectiveness of offering different levels ($10 and $20) of a promised incentive. This 
experiment also varied whether the household was notified of the promised incentive in the 
letter enclosed in the screener mailings. 

The effect of the incentive and also the effect of prenotification (given a particular 
incentive level) were each tested using chi-square tests. This approach was used because 
at these levels, there was no a priori belief that there should be an interaction effect and 
there is more power to test for the separate main effects. That is, the issues addressed 
through this set of tests are, first, whether the incentive has an effect and, second, if an 
incentive is used, should it be communicated to the respondent with a prenotification? For 
the national sample, the level of incentive was found to have a statistically significant effect 
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on the screener response rate and on the percentage of respondents providing a phone 
number in the screener, but it did not affect the extended interview response rate 
(table B-14). For those designated to receive an incentive for completing the extended 
interview, notification of the incentive was found to have a significant effect on both the 
screener and extended interview response rates and on the percentage of respondents 
providing a phone number in the screener. Post-hoc analyses to examine individual 
differences were not conducted. 

B.6.3 Differences in characteristics of sampled adults according to availability of 
telephone numbers 

Table B-15 provides statistics about the availability of telephone numbers. At the 
time of sample selection, sampled addresses were reverse-matched to directory listings by 
a vendor in an effort to obtain phone numbers. As a result of this reverse-match, phone 
numbers were available for 63 percent of sampled addresses and 67 percent of sampled 
adults. The ATES screener also requested a phone number for each enumerated adult. 
(See attachment B-2 for an illustration of the screener.) Taking into account both the 
vendor-matched phone numbers and the numbers provided in the screener, phone 
numbers were available for 96 percent of sampled adults. 

While requesting a phone number for each enumerated adult clearly boosted the 
proportion of sampled adults with whom the extended interview could be attempted, an 
additional question is whether this also changed the characteristics of the sample and, in 
doing so, potentially reduced bias. To examine this, tabulations of various characteristics of 
the sampled adults reported in the screener by source(s) of phone number(s) are given in 
table B-16. 

Table B-17 is similar to table B-16, except that the source(s) of the phone number 
have been collapsed to more clearly demonstrate any changes resulting from obtaining a 
phone number in the screener that was not previously available for the sampled adult. As 
shown in table B-17, asking for a phone number in the screener resulted in the ability to 
attempt an extended interview with a set of adults who were younger and less highly 
educated than those for whom the vendor phone number was used to attempt the extended 
interview. (The table also indicates a much higher proportion of screener-provided phone 
numbers that were cell/wireless, but this was to be expected since the vendor matches are, 
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by design, virtually all landline numbers.) For each of these characteristics, including the 
cases for which the screener phone number was used to attempt the extended interview 
resulted in a set of sampled adults that more closely resembles the population distribution. 
(Estimates of the population distribution are from the 3-year 2006-08 ACS.) 
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Table B-13. Cumulative number and percentage of sampled adults who responded to the screener in the ATES Pilot 
Study, by screener mailing wave and characteristics: 2010-11 


Initial response (wave 1 After first screener follow-up (wave After second screener follow-up 

respondents only) 1 1 and 2 respondents) 1 (wave 1 , 2, and 3 respondents) 1 


Screener characteristic 

ACS percentage 

Number of 
respondents 

Percent of 
respondents 

Number of 
respondents 

Percent of 
respondents 

Number of 
respondents 

Percent of 
respondents 

Age 








18 to 29 years old 

21.3 

758 

13.6 

954 

13.8 

1,204 

14.3 

30 to 49 years old 

38.2 

1,703 

30.6 

2,133 

30.8 

2,654 

31.5 

50 years old and over 

40.6 

3,102 

55.8 

3,847 

55.5 

4,579 

54.3 

Chi-square 


28.58 


25.28 




p value 


<0.0001 


<0.0001 




Sex 








Male 

48.3 

2,561 

45.7 

3,130 

45.3 

3,779 

45.0 

Female 

51.7 

3,037 

54.3 

3,781 

54.7 

4,624 

55.0 

Chi-square 


4.12 


1.32 




p value 


0.13 


0.52 




Education 








Less than HS 

15.3 

400 

7.1 

506 

7.3 

656 

7.8 

HS or equivalent 

51.8 

2,681 

47.8 

3,353 

48.5 

4,118 

48.9 

Some college 

33.0 

2,528 

45.1 

3,059 

44.2 

3,641 

43.3 

Chi-square 


29.57 


26.00 




p value 


<0.0001 


<0.0001 




Type of phone 








Landline 


2,925 

62.4 

3,541 

62.1 

4,198 

61.5 

Cell/wireless 


1,752 

37.4 

2,148 

37.7 

2,619 

38.3 

Both checked 


7 

0.1 

11 

0.2 

14 

0.2 

Chi-square 


8.73 


8.04 




p value 


0.07 


0.09 





1 Counts do not sum to totals and differ across characteristics, due to the exclusion of cases with missing values for the particular characteristic. 

NOTE: ACS is the American Community Survey. The chi-square tests are based on comparisons of individual waves of ATES respondents. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. U.S. Department of 
Commerce, U.S. Census Bureau, American Community Survey (ACS), 2005-09. 
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Table B-14. Number of screener and extended interview respondents and response rates and number and percentage 
of respondents providing a telephone number in the screener of the ATES Pilot Study, by type of 
incentive treatment: 2010-11 


Incentive/notification 

treatment 

Number of 
screener 
respondents 

Screener 
response rate 

Standard 

error 

Number of 
extended 
interview 
respondents 

Extended 
interview 
response rate 
(weighted) 

Standard 

error 

Among screener respondents 

Number Percent 

providing providing 

phone number phone number 

Standard 

error 

$0 

1,650 

50.3 

0.01 

689 

42.2 

1.33 

1,437 

87.2 

0.01 

S10— notified 

2,566 

51.0 

0.01 

1,177 

45.5 

1.14 

2,318 

90.4 

0.01 

$10— not notified 

802 

49.6 

0.02 

326 

40.0 

1.78 

703 

87.8 

0.01 

$20— notified 

2,632 

52.5 

0.01 

1,177 

44.2 

0.93 

2,395 

91.1 

0.01 

$20— not notified 

828 

51.3 

0.02 

361 

43.7 

1.99 

718 

86.7 

0.01 

$0 

1,650 

50.3 

0.01 

689 

42.2 

1.33 

1,437 

87.2 

0.01 

$10 

3,368 

50.7 

0.01 

1,503 

44.2 

0.97 

3,201 

89.8 

0.01 

$20 

3,460 

52.2 

0.01 

1,538 

44.1 

0.87 

3,113 

90.0 

0.01 

Chi-square 

786.81 



1.60 



10.39 



p value 

<0.0001 



0.45 



0.006 



$10 or $20— notified 

5,198 

51.8 

0.01 

2,354 

44.9 

0.74 

4,713 

90.7 

0.00 

$1 0 or $20— not notified 

1,630 

50.5 

0.01 

687 

41.9 

1.29 

1,421 

87.2 

0.01 

Chi-square 

461.11 



3.89 



16.81 



p value 

<0.0001 



0.048 



<0.0001 




NOTE: Screener response rates and the percentage of respondents providing a phone number in the screener are unweighted estimates and have simple random sampling 
standard errors, while extended interview response rates are weighted estimates and have standard errors that reflect the complex sample design and estimation procedures. The 
chi-square tests are testing for association between the two treatment main effects (incentives and notification, separately) and screener response, extended interview response, 
and the proportion providing a phone number. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. 
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Table B-15. Telephone number availability of sampled adults in ATES Pilot Study, by 
source of phone number: 2010-11 


Source of phone number 

Number of 
sampled adults 

Percent of 
sampled adults 

Total 

8,471 

100.0 

No telephone number available 

315 

3.7 

Vendor phone number, no screener phone number 

585 

6.9 

Screener phone number same as vendor phone number 

2,947 

34.8 

Screener phone number, different from vendor phone number 

2,305 

27.2 

Screener phone number, no vendor phone number 

2,319 

27.4 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 
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Table B-16. Comparison of screener characteristics of sampled adults in ATES Pilot Study (national sample only), by 
source of telephone numbers: 2010-11 



Percent 

Only vendor phone available 

Only screener phone 
available 

Screener phone is identical to 
vendor phone 

Screener phone is different 
from vendor phone 

Screener 

characteristic 

of 

population 

(ACS) 

Stand- 

ard 

Number Percent error 

Stand- 

ard 

Number Percent error 

Stand- 

ard 

Number Percent error 

Stand- 

ard 

Number Percent error 


Total 


585 

6.3 

0.31 

2,319 

28.0 

0.49 

2,947 

35.6 

0.53 

2,305 

26.5 

0.53 

Age 














18 to 29 years 
old 

21.3 

44 

10.1 

1.66 

424 

22.0 

0.98 

227 

11.3 

0.56 

467 

23.1 

1.00 

30 to 49 years 
old 

38.2 

138 

26.6 

1.94 

925 

38.9 

1.08 

688 

24.7 

0.85 

796 

34.4 

1.07 

50 years old 
and over 

40.6 

392 

63.4 

2.22 

964 

39.1 

0.90 

2,027 

64.0 

1.06 

1,034 

42.5 

1.24 

Sex 














Male 

48.3 

222 

42.3 

2.27 

1,081 

48.4 

1.25 

1,290 

46.8 

1.16 

1,053 

48.0 

1.01 

Female 

51.7 

352 

57.7 

2.27 

1,219 

51.7 

1.25 

1,639 

53.2 

1.16 

1,237 

52.0 

1.01 

Education 














Less than HS 

15.3 

44 

6.9 

1.13 

195 

9.1 

0.64 

200 

6.8 

0.50 

190 

8.5 

0.58 

HS orequiv. 

51.8 

311 

54.1 

2.06 

1,094 

48.4 

0.97 

1,461 

49.9 

1.00 

1,091 

49.1 

1.15 

Some college 

33.0 

220 

39.1 

1.82 

1,014 

42.5 

1.00 

1,277 

43.3 

0.91 

1,005 

42.5 

1.16 

Type of phone 














Landline 

— 

54 

72.6 

6.05 

871 

42.7 

1.24 

2,561 

99.2 

0.20 

698 

32.6 

1.00 

Cell/wireless 

— 

14 

21.7 

5.02 

1,177 

57.2 

1.26 

16 

0.5 

0.14 

1,386 

67.3 

1.00 

Both checked 

— 

3 

5.7 

3.46 

2 

0.1 

0.07 

7 

0.3 

0.14 

2 

0.1 

0.04 


— Not available. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11; and U.S. Department of 
Commerce, Census Bureau, American Community Survey (ACS), 2005-09. 
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Table B-17. Comparison of screener characteristics of sampled adults in ATES Pilot Study (national sample 
only), by source of telephone numbers (collapsed): 2010-11 


Only vendor phone available or Only screener phone available or 

Any valid phone number vendor phone is identical to vendor phone is different from 

available screener phone screener phone 


Screener 

characteristic 

ACS 

Proportion 

Number 

Percent 

Standard 

error 

Number 

Percent 

Standard 

error 

Number 

Percent 

Standard 

error 

Total 

Age 

18 to 29 years 


8,156 

100.0 

t 

3,532 

43.5 

0.53 

4,624 

56.5 

0.53 

old 

30 to 49 years 

21.3 

1,162 

17.6 

0.45 

271 

11.1 

0.54 

891 

22.6 

0.68 

old 

50 years old and 

38.2 

2,547 

31.6 

0.50 

826 

25.0 

0.76 

1,721 

36.7 

0.72 

over 

Chi-square 
p value 
Sex 

40.6 

4,417 

50.8 

0.62 

2,419 

439.60 

<0.0001 

63.9 

0.92 

1,998 

40.8 

0.73 

Male 

48.3 

3,646 

47.3 

0.70 

1,512 

46.1 

1.12 

2,134 

48.2 

0.87 

Female 
Chi-square 
p value 
Education 

51.7 

4,447 

52.7 

0.70 

1,991 

2.28 

0.131 

53.9 

1.12 

2,456 

51.8 

0.87 

Less than HS 

15.3 

629 

7.9 

0.31 

244 

6.8 

0.44 

385 

8.8 

0.42 

HS or equivalent 

51.8 

3,957 

49.5 

0.53 

1,772 

50.5 

0.85 

2,185 

48.7 

0.69 

Some college 

33.0 

3,516 

42.6 

0.54 

1,497 

42.7 

0.79 

2,019 

42.5 

0.75 


Chi-square 10.98 

p value 0.004 

See notes at end of table. 
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Table B-17. Comparison of screener characteristics of sampled adults in ATES Pilot Study (national sample 
only), by source of telephone numbers (collapsed): 2010-11 — Continued 


Only vendor phone available or Only screener phone available or 

Any valid phone number vendor phone is identical to vendor phone is different from 

available screener phone screener phone 


Screener 

characteristic 

ACS 

Proportion 

Number: 

Percent 

Standard 

error 

Number 

Percent 

Standard 

error 

Number 

Percent 

Standard 

error 

Type of phone 











Landline 

— 

4,184 

61.9 

0.58 

2,615 

98.6 

0.24 

1,569 

37.7 

0.80 

Cell/wireless 

— 

2,593 

37.9 

0.58 

30 

1.0 

0.19 

2,563 

62.2 

0.81 

Both checked 

— 

14 

0.2 

0.06 

10 

0.4 

0.15 

4 

0.1 

0.04 


Chi-square 2,113.67 

p value <0.0001 

— Not available, 
t Not applicable. 

NOTE: The Chi-square test is testing for association between the given screener characteristic and phone number source (“Only vendor phone available or vendor 
phone is identical to screener phone”/”Only screener phone available or vendor phone is different from screener phone”). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11; and U.S. 
Department of Commerce, Census Bureau, American Community Survey (ACS), 2006-09. 
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Table B-18 presents characteristics of extended interview respondents according 
to the source(s) of the phone number(s); the tabulations include both base-weighted 
distributions and final-weighted distributions. Table B-19 shows the same characteristics 
with chi-square test statistics and p values based on the final-weighted distributions. The 
estimates in table B-19 indicate that asking for a phone number in the screener resulted in 
the ability to complete an extended interview with a set of adults who were younger, were 
more likely to be Hispanic or Black, had an income distribution that was shifted downward, 
were more likely to have worked in the last 12 months, were less likely to own their homes, 
and were more likely to live in the South or West than those for whom the vendor phone 
number was used to complete the extended interview. For each of these characteristics 
other than income, including the cases for which the screener phone number was used to 
attempt the extended interview resulted in completed interviews with a set of sampled 
adults that more closely resembles the population distribution. (Estimates of the population 
distribution are from the 5-year 2005-09 ACS.) 

B.6.4 Unit response rates by characteristics of the address 

The ABS includes various characteristics of the address, maintained by the USPS. 
These characteristics include an indicator of whether the address is seasonal and an 
indicator of whether the address is vacant. Another characteristic included in the file is 
whether the address is a drop point — a single delivery point that services more than one 
residential unit (with no separate USPS delivery to each individual unit). Drop points are 
problematic because, without a name, there is no way to specify delivery to a particular unit. 

In the ATES Pilot Study, screeners were mailed to each of these types of addresses; 
none were specifically excluded. However, because it is likely that address eligibility rates 
and response rates may differ according to these characteristics, it is worthwhile to consider 
them. Table B-20 gives address eligibility, screener response rates, and extended interview 
response rates by characteristics of the address, and table B-21 contains tabulations on the 
source of phone numbers by characteristics of the address (for the subset of sampled 
adults for whom a phone number was available). 
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Table B-18. Comparison of characteristics of extended survey respondents in ATES Pilot Study (national 
sample only), by source of telephone numbers (collapsed): 2010-11 



Only vendor phone available or vendor phone is 
identical to screener phone 

Any phone number available 



Base 


Final 


Base 


Final 


Base weighted 

Final 

weighted 

Base 

weighted 

Final 

weighted 

Respondent 

weighted (standard 

weighted 

(standard 

weighted 

(standard 

weighted 

(standard 

characteristic 

Number percent error ) 

percent 

error) 

Number percent 

error) 

percent 

error) 


Age 


18 to 29 years old 

74 

7.1 

0.69 

11.4 

1.03 

479 

15.4 

0.72 

21.3 

0.00 

30 to 49 years old 

325 

23.0 

1.36 

29.6 

1.24 

1,111 

30.8 

0.89 

38.2 

0.00 

50 years old and over 

1,154 

69.9 

1.51 

59.1 

1.35 

2,140 

53.9 

0.97 

40.6 

0.00 

Sex 

Male 

650 

45.4 

1.39 

48.9 

1.24 

1,631 

45.6 

0.94 

48.3 

0.00 

Female 

903 

54.6 

1.39 

51.1 

1.24 

2,099 

54.4 

0.94 

51.7 

0.00 

Education 

Less than HS 

140 

8.5 

0.75 

13.9 

1.10 

337 

8.9 

0.48 

15.3 

0.00 

HS orequiv. 

705 

45.5 

1.47 

53.9 

1.14 

1,602 

43.3 

0.86 

51.8 

0.00 

Some college 

708 

45.9 

1.42 

32.2 

0.98 

1,791 

47.8 

0.92 

33.0 

0.00 

Race/ethnicity 1 

Black 

95 

5.9 

0.63 

7.2 

0.78 

346 

8.7 

0.48 

10.6 

0.18 

Hispanic 

41 

2.7 

0.45 

5.2 

0.91 

236 

6.9 

0.43 

13.2 

0.00 

Other 

1,384 

89.1 

0.90 

85.5 

1.43 

3,044 

81.7 

0.66 

73.2 

0.28 

Two or more races 

33 

2.2 

0.38 

2.1 

0.36 

104 

2.8 

0.29 

2.9 

0.35 


See notes at end of table. 
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Table B-18. Comparison of characteristics of extended survey respondents in ATES Pilot Study (national 
sample only), by source of telephone numbers (collapsed): 2010-11 — Continued 


Only vendor phone available or vendor phone is 

identical to screener phone Any phone number available 


Respondent 

characteristic 

Number 

Base 

weighted 

percent 

Base 
weighted 
(standard 
error ) 

Final 

weighted 

percent 

Final 

weighted 

(standard 

error) 

Number 

Base 

weighted 

percent 

Base 

weighted 

(standard 

error) 

Final 

weighted 

percent 

Final 

weighted 

(standard 

error) 

Income 











$0— 25K 

394 

21.0 

1.01 

25.0 

1.32 

998 

23.3 

0.69 

28.0 

0.79 

$25,001— 75K 

702 

45.7 

1.33 

45.6 

1.55 

1,659 

44.6 

0.71 

45.2 

0.89 

$75K+ 

457 

33.3 

1.20 

29.4 

1.20 

1,073 

32.0 

0.63 

26.8 

0.72 

Employment 

status 











Worked in the 
last 12 months 

806 

56.9 

1.20 

58.4 

1.31 

2,344 

65.6 

0.76 

66.8 

0.84 

Did not work in 
the last 12 
months 

747 

43.1 

1.20 

41.7 

1.31 

1,386 

34.4 

0.76 

33.2 

0.84 

Home tenure 











Own home 

1,297 

83.3 

0.99 

78.7 

1.32 

2,621 

70.6 

0.80 

63.4 

0.94 

Rent home 

160 

9.3 

0.75 

11.4 

0.99 

817 

19.8 

0.75 

24.4 

0.91 

Other 

96 

7.5 

0.75 

10.0 

1.10 

292 

9.6 

0.49 

12.3 

0.66 


See notes at end of table. 
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Table B-18. Comparison of characteristics of extended survey respondents in ATES Pilot Study (national 
sample only), by source of telephone numbers (collapsed): 2010-11 — Continued 


Only vendor phone available or vendor phone is 

identical to screener phone Any phone number available 

Base Final Base Final 

Base weighted Final weighted Base weighted Final weighted 

Respondent weighted (standard weighted (standard weighted (standard weighted (standard 


characteristic 

Number 

percent 

error ) 

percent 

error) 

Number 

percent 

error) 

percent 

error) 

Census region 2 











Northeast 

338 

21.9 

1.09 

23.2 

1.00 

676 

18.4 

0.56 

18.5 

0.00 

Midwest 

446 

28.3 

1.18 

25.2 

1.14 

950 

25.2 

0.62 

22.0 

0.00 

South 

525 

33.4 

1.13 

35.7 

1.14 

1,303 

34.1 

0.63 

36.5 

0.00 

West 

244 

16.4 

1.03 

15.9 

1.00 

801 

22.4 

0.59 

23.0 

0.00 


1 Black includes African American, Hispanic includes Latino, Other includes all other races, and two or more races includes non-Hispanics reporting more than 
one race. 

2 The following states and the District of Columbia are in each Census region: Northeast: CT, MA, ME, NH, NJ, NY, PA, Rl, VT; South: AL, AR, DC, DE, FL, 
GA, KY, LA, MD, MS, NC, OK, SC, TN, TX, VA, WV; Midwest: IA, IL, IN, KS, Ml, MN, MO, ND, NE, OH, SD, Wl; West: AK, AZ, CA, CO, HI, ID, MT, NV, NM, 
OR, UT, WA, WY. 

NOTE: Chi-square statistics test vendor phone availability with screener phone availability for the given characteristics. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11; and 
U.S. Department of Commerce, Census Bureau, American Community Survey (ACS), 2006-08. 
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Table B-19. Comparison of characteristics of extended survey respondents in ATES Pilot Study 
(national sample only), by source of telephone numbers: 2010-11 


Respondent 

characteristic 

ACS 

proportion 

Only vendor phone available or vendor 
phone is identical to screener phone 

Final weighted Final weighted 
Number percent (standard error) 

Only screener phone available or vendor 
phone is different from screener phone 

Final weighted Final weighted 

Number percent (standard error) 

Age 








18 to 29 years old 

21.3 

74 

11.4 

1.03 

405 

27.0 

0.55 

30 to 49 years old 

38.2 

325 

29.6 

1.24 

786 

43.2 

0.68 

50 years old and over 

40.6 

1,154 

59.1 

1.35 

986 

29.7 

0.61 

Chi-square 


271.53 






p value 


<0.0001 






Sex 








Male 

48.3 

650 

48.9 

1.24 

981 

47.9 

0.73 

Female 

51.7 

903 

51.1 

1.24 

1,196 

52.1 

0.73 

Chi-square 


0.26 






p value 


0.611 






Education 








Less than HS 

15.3 

140 

13.9 

1.10 

197 

16.1 

0.64 

HS orequiv. 

51.8 

705 

53.9 

1.14 

897 

50.5 

0.68 

Some college 

33.0 

708 

32.2 

0.98 

1,083 

33.4 

0.58 

Chi-square 


3.52 






p value 


0.172 






Race/ethnicity 1 








Black 

11.1 

95 

7.2 

0.78 

251 

12.7 

0.54 

Hispanic 

13.2 

41 

5.2 

0.91 

195 

18.0 

0.53 

Other 

74.6 

1,384 

85.5 

1.43 

1,660 

66.0 

0.73 

Two or more races 

1.1 

33 

2.1 

0.36 

71 

3.4 

0.53 


Chi-square 126.88 

p value <0.0001 

See notes at end of table. 


B-50 







Table B-19. Comparison of characteristics of extended survey respondents in ATES Pilot Study 
(national sample only), by source of telephone numbers: 2010-11 — Continued 




Only vendor phone available or vendor 

Only screener phone available or vendor 



phone is identical to screener phone 

phone is different from screener phone 

Respondent 

ACS 

Final weighted Final weighted 

Final weighted Final weighted 

characteristic 

proportion 

Number percent (standard error) 

Number percent (standard error) 


Income 


$0— 25K 

16.1 

394 

25.0 

1.32 

604 

29.9 

1.07 

$25,001— 75K 

43.6 

702 

45.6 

1.55 

957 

44.9 

1.15 

$75K+ 

33.2 

457 

29.4 

1.20 

616 

25.3 

0.98 

Chi-square 


10.22 






p value 


0.006 






Employment 

status 








Worked in the 
last 12 months 

64.8 

806 

58.4 

1.31 

1,538 

71.7 

1.27 

Did not work in 
the last 12 
months 

34.8 

747 

41.7 

1.31 

639 

28.3 

1.27 

Chi-square 


45.39 






p value 


<0.0001 






Home tenure 








Own home 2 

67.1 

1,297 

78.7 

1.32 

1,324 

54.4 

1.43 

Rent home 

32.9 

160 

11.4 

0.99 

657 

32.0 

1.32 

Other 1 

— 

96 

10.0 

1.10 

196 

10.0 

1.10 

Chi-square 


130.53 






p value 


<0.0001 







See notes at end of table. 
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Table B-19. Comparison of characteristics of extended survey respondents in ATES Pilot Study 
(national sample only), by source of telephone numbers: 2010-11 — Continued 




Only vendor phone available or vendor phone 
is identical to screener phone 

Only screener phone available or vendor 
phone is different from screener phone 

Respondent 

characteristic 

ACS 

proportion 

Final weighted Final weighted 
Number percent (standard error) 

Final weighted Final weighted 
Number percent (standard error) 


Census region 3 


Northeast 

18.5 

338 

23.2 

1.00 

338 

15.7 

0.57 

Midwest 

22.0 

446 

25.2 

1.14 

504 

20.2 

0.67 

South 

36.5 

525 

35.7 

1.14 

778 

36.9 

0.67 

West 

23.0 

244 

15.9 

1.00 

557 

27.2 

0.58 


Chi-square 61.79 

p value <0.0001 

— Not available. 

1 Black includes African American, Hispanic includes Latino, Other includes all other races, and two or more races includes non-Hispanics reporting 
more than one race. 

2 The ACS combines the “Own home” and “Other” categories. 

3 The following states and the District of Columbia are in each Census region: Northeast: CT, MA, ME, NH, NJ, NY, PA, Rl, VT; South: AL, AR, DC, 
DE, FL, GA, KY, LA, MD, MS, NC, OK, SC, TN, TX, VA, WV; Midwest: IA, IL, IN, KS, Ml, MN, MO, ND, NE, OH, SD, Wl; West: AK, AZ, CA, CO, HI, 
ID, MT, NV, NM, OR, UT, WA, WY. 

NOTE: The Chi-square test is testing for association between the given characteristic and phone number source (“Only vendor phone available or 
vendor phone is identical to screener phoneTOnly screener phone available or vendor phone is different from screener phone”). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 
2010-11; and U.S. Department of Commerce, Census Bureau, American Community Survey (ACS), 2005-09. 
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Table B-20. Screener and extended survey response rates in ATES Pilot Study (national sample only), by address 
characteristics: 2010-11 



Total sampled addresses 

Address eligibility 1 


Screener response 

Extended interview response 

Characteristic of address 

Number 

Percent 

Number Eligibility rate 

Number 

Response rate 

Number 

Response rate 

Seasonal address 

No 

18,604 

99.2 

16,488 

88.6 

8,442 

51.2 

3,716 

43.8 

Yes 

146 

0.8 

64 

43.8 

36 

56.3 

14 

39.7 

Vacant 

No 

17,595 

93.8 

16,267 

92.5 

8,360 

51.4 

3,672 

43.7 

Yes 

1,155 

6.2 

285 

24.7 

118 

41.4 

58 

51.0 

Drop point 

No 

18,584 

99.1 

16,398 

88.2 

8,410 

51.3 

3,694 

43.8 

Augmented drop point 2 

48 

0.3 

47 

97.9 

25 

53.2 

16 

55.0 

Yes 

118 

0.6 

107 

90.7 

43 

40.2 

20 

45.8 


1 An address was considered eligible if a screener sent to that address was not returned as undeliverable. 

2 Augmented drop points are drop point addresses in which the vendor was able to append specific unit information by linking to other (non-USPS) data sources. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. 
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Table B-21. Source of telephone numbers for sampled adults for whom a telephone number was available in ATES 
Pilot Study (national sample only), by address characteristics: 2010-11 



Screener respondents 


Only vendor phone available or vendor 
phone is identical to screener phone 

Only screener phone available or vendor 
phone is different from screener phone 

Characteristic of address 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Total 

8,478 

100.0 

3,532 

43.5 

4,624 

56.5 

Seasonal address 

No 

8,442 

99.6 

3,519 

43.3 

4,601 

56.7 

Yes 

36 

0.4 

13 

36.1 

23 

63.9 

Vacant 

No 

8,360 

98.6 

3,524 

43.8 

4,520 

56.2 

Yes 

118 

1.4 

8 

7.2 

104 

92.9 

Drop point 

No 

8,410 

99.2 

3,497 

43.3 

4,591 

56.8 

Augmented drop point 1 

25 

0.3 

9 

36.0 

16 

64.0 

Yes 

43 

0.5 

26 

60.5 

17 

39.6 


1 Augmented drop points are drop point addresses in which the vendor was able to append specific unit information by linking to other (non-USPS) data sources. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. 
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B.7 Discussion 


Although the ATES Pilot Study was designed to facilitate analyses aimed at 
determining the best ways to capture information on the attainment of education certificates 
and of industry-based licenses and certifications, it has important implications for future 
studies of the educational attainment and credentials of adults. A key finding is that, in the 
face of declining unit response rates to telephone surveys and declining coverage of 
landline RDD surveys, the ABS approach used for the ATES Pilot Study was generally 
successful. The screener attained a unit response rate of 51 percent (for the national 
sample) and, after screening, telephone numbers were available for 96 percent of sampled 
adults. These results bode well for a general population household study aiming to reach 
randomly selected adults to conduct an extended interview by telephone. However, it 
should be noted that factors such as sponsorship, topic salience, the length of the data 
collection period, the number and sequenced nature of the screener follow-up mailings, the 
use of incentives, specifics of the survey materials, and the use of FedEx delivery service 
for the final screener follow-up mailing may each have had important effects on the ability to 
attain high unit response rates. Additionally, strategies not used in the Pilot Study, such as 
offering a bilingual or dual (English and Spanish) screener, and offering a larger screener 
incentive in the initial screener mailing, might result in higher unit response rates. 

The unit response rate to the extended interview (42 percent for the national sample 
in the ATES Pilot Study) is lower than might otherwise be attained, due to the following 
considerations: 

• The primary foci of the ATES Pilot Study were sample yield and timeliness, not 
response rate. As a result, the data collection efforts were in some cases curtailed 
(e.g., limited telephone nonresponse follow-up). 

• The extended interview was attempted only in English. (About 5 percent of 
nonresponse to the extended interview was the result of the case being coded a final 
language problem.) 

• The incentive for completing the extended interview was promised in the screener 
letter (for the cases randomly designated to be notified in advance) and at the 
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beginning of the extended interview call. Offering a prepaid (rather than promised) 
incentive for completing the extended interview might have increased cooperation. 
However, the large prepaid incentives required to significantly boost response would 
also significantly increase cost and extend the length of the survey period (for the 
telephone interview) because an additional mailing is required. 

• Refusal conversion did not begin until late in the extended interview data collection 
period. 

• For cases in which the phone number loaded for the extended interview was not 
working, was a business number, or could not be used to get in touch with the 
sampled adult, no attempt was made to reach the adult using any other phone 
number that might have been available. 

In addition to these factors, mode considerations are known to be important drivers 
of response rates. For example, in a large-scale pilot study conducted for the National 
Household Education Surveys Program (NHES) in 2009, the telephone was shown to be a 
much less effective mode for nonresponse follow-up than mail (Montaquila et al. 2010). 
Although it was necessary to conduct the extended interview in the ATES Pilot Study by 
telephone only, the use of mail in place of, or in combination with, the telephone (and 
possibly other modes) would likely have an effect on response rates. 

The ABS sampling frame used to select the national sample of addresses for the 
ATES Pilot Study covers virtually all residential addresses in the United States. By 
comparison, it is currently estimated that only 71 percent of households (73 percent of 
adults) have a landline phone number (Blumberg and Luke 2010), and a sizable proportion 
of these numbers are not included in the standard list-assisted landline RDD sampling 
frames. And it has been well established that characteristics of persons in cell-only 
households differ from those in households with landline phones (Blumberg and Luke 
2010). Although coverage may be increased by using a dual-frame (landline and cell) RDD 
approach, there are many operational issues and substantial cost implications with doing 
so, and the differential response of dual-user (landline and cell) households according to the 
frame from which they are sampled presents some challenging estimation issues (AAPOR 
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2010). The mail-based ABS methodology used in the first phase of data collection for the 
ATES Pilot Study provides a viable alternative to RDD. 
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A Verifier operator reviewed the responses in each form in a multistep process in a 
module of Cardiff TeleForm called Verifier. Verifier operators were provided with written 
rules for the Adult Training and Education Survey (ATES) Pilot Study indicating how to 
handle values that were out of range or responses that did not conform to the data 
specifications. If a response did not conform to the data specifications and a rule did not 
exist for handling the response, then the Verifier operator recorded in the Data Decision Log 
(DDL) the ID number, the field name, the response (as written on the paper copy of the 
form), and the response as it appeared in Verifier. Once verification was complete, the form 
was automatically exported to the database. 

TeleForm Verifier rules were written to enable the Verifier operator to review any 
data fields flagged in the scan process and to either make the necessary clarification and/or 
flag the item(s) for further review. Project-specified Verifier rules used for the ATES Pilot 
Study are as follows (these are written as they were provided to the Verifier operators): 

• Responses should appear exactly as written, except where indicated in Exhibit B-1 
below. 

• Do not scan blank forms. If a survey is blank, then notify the supervisor and DO NOT 
VERIFY. Check that the correct disposition code MN1 is recorded in the receipt 
system. Forms that are returned with only Question 1 answered (NUM18Up and no 
other data) should be treated as blanks (coded MN1). Forms with Q1 blank, but 
other data are entered are processed. 

• If more than one response is provided but Verifier will only accept one response, 
then leave the field blank and RECORD THIS IN YOUR DATA DECISION LOG. 
Please see Exhibit B-1 below for exceptions to this rule . 

• If the respondent marks between two boxes, then leave the question blank and 

RECORD THIS IN YOUR DATA DECISION LOG. 

• If the respondent writes NA, N/A, not applicable, DK, don’t know, unknown, or 
something similar and no boxes are checked, then leave the question blank and 
RECORD THIS IN YOUR DATA DECISION LOG. You do not need to record this in 
your entry log if there are no data in the column, i.e., there are no additional adults in 
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the household and the respondent writes NA at the top of the column, or across the 
page. 

• If you have trouble reading the handwriting, then alert your supervisor. If a decision 
cannot be made, then leave blank. RECORD THIS IN YOUR DATA DECISION 
LOG. 

• If a form has been completed using something that will not be picked up in scanning 
(e.g., light pencil, gold glitter pens), or a form is torn or too damaged to scan, alert 
your supervisor. Data should be carefully copied to a clean form and QCed by 
another team member. Record this in your entry log, and file both copies together. 

• Enter numbers exactly as written. If Verifier gives you a field validation message 
saying that the value is out of range, then enter the number as written and click 
“Accept value and set field status to OK.” 

• In the unlikely event that we received a form from the call center, process (receipt- 
scan) as usual. 

• Duplicate forms should be scanned. 

• For comments in the margins, process only scannable data. 

• In the case of a barcode that has been cut off, check if a phone number has been 
provided. If there is a phone number, send it to the data manager for follow up with 
project staff. If there is no phone number, file. 

• Rounding Rule: 0.5 and up, round up; below 0.5, round down. 


B-63 



Exhibit B-1. Additional Verification Procedures for Screener Questionnaire Items 


Question Number 

Variable Name 

Scenario 

Instructions 

Q1 


NUM18UP 

Response is incongruent (e.g., 
under 18 years old). 

Process as is. 

Q3 


AGE1 

AGE2 

AGE3 

AGE4 

AGE is less than 18 

Leave as is (LAI) and process. It is not 
necessary to record this in the entry log. 

Q4 


SEX1 

SEX2 

SEX3 

SEX4 

More than one box is marked. 

Leave blank. 

Q5 


GRADE1 

GRADE2 

GRADE3 

GRADE4 

Multiple responses checked. 

Record highest response. 

Q6 


PH0NE1 

PH0NE2 

PH0NE3 

PH0NE4 

Incomplete phone number. 

Process. 

Q6 


SAMEPH0NE2 

SAMEPH0NE3 

SAMEPH0NE4 

Box checked and different phone 
number provided. 

Leave as is and process. 

Q7 


TYPEPH0NE1 

TYPEPH0NE2 

TYPEPH0NE3 

TYPEPH0NE4 

Both landline and cell phone 
checked. 

Teleform should not stop. The value ‘9’ 
will be written to the database for this 
variable. 

Q7 


TYPEPH0NE1 

TYPEPH0NE2 

TYPEPH0NE3 

TYPEPH0NE4 

Neither box is checked, and 
respondent writes home phone 
or similar next to box 

Check home phone box 
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ATTACHMENT B-2: 

Screener Questionnaire, Cover Letters, 
and Reminder Postcard Examples 
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Commonly Asked Questions 


Q: How did you net my address? 

A Your ackYoss was randomly selected from among al of me residential mailing addresses « the nation II was 
selected using scientific sampling methods to represent other households n the U S 

9; Why »9« mK rw?r? n«W"» mirt IrUnlre tWtniOT In Bitt 9«twwnnrt? 

A The purpose of this questionnaire is to find out if anyone n your household is eligible for the next stage of the 
sixvey if so . we wd contact a member of your household to ask about their educational experiences 

9 Why ' laKt Ml 'n '. h ‘t twit? P? ' hart 19 »? put? 

A You represent thousands ot other households bke yours, and you cannot be replaced Your answers and 
opimons are very important to the success of this survey Your participation is voluntary You may choose not to 
answer any or all questions in this survey There are no penalties should you choose Qg| to participate n the 
survey In order for the survey to be representative a is mportant that you complete and return the questionnaire 

Q: How will th« intomulion I provide bo iiwd? 

A: Your responses wi« be combined with those of others to produce statistical summaries and reports Your 
individual data wd not be reported Your answers may be used only for statistical purposes and may not be 
disclosed or used, m identifiable form for any other purpose except as required by law [Education Sciences 
Reform Act of 2002 (ESRA 20021 Public Law 107-279. Section 183| 


A On average. < should lake 3 mnutes for you to respond, including tho tmo tor reviewing instructions, and 
completing and reviewing the collection of information Send comments rngarrjng this burden estimate or any 
other aspect of tt*s collection of information including suggestions for reducing this burden, to The Adult 
Trairtng and Education Survey National Center for Education Statistics. U S Department of Education 1990 K 
Street NW. Washington. DC 20006-5650 Do not return the completed form to thrs address 


Q: Should i mtliKH mv tonKUuahttf wt>o .» nrn n »chool? SNould I mcludt mv moihtr who .» vu'llno fo< ih» 


A include adults age 18 and older who are temporarily away from homo (for example, attending college or 
unrversity) 4 they have no other permanent home Do not include gucstsVisitor* who have another permanent 
home 


9 W'ii il^ttlflglUaiBjB.nigg.aaa.4 I" tiana!iglfl2 

A II there are more than four adults m your household please call 1-868-627-5304 for instructions about how to 
complete the form 

9- Who.3 je°n»o-mfl »ury»l2 j$ .ih_l? »ury»ii COfldjjCltd.bt l_h« Federal Goy«n>m»nt? 

A The National Center tor Education Statistics within the Department of Education, is authorized to conduct this 
survey (Section 9543. 20 US Code) Wes tat has been contracted to conduct this survey This survey has been 
approved by the Office of Management and Budget, the office that reviews all tederaly sponsored surveys The 
approval number assigned to ths survey is 16500603 You may send any comments about this survey, 
ndudng its length to the Federal Government Write to The Adult Training and Education Survey. National 
Center for Education Statistics. U S Department or Education 1990 K Street NW. Washington. DC 20006-5650 
You may send e-mai to ATE S I nfo@ Westat com 

Q: Who is Westat? 

A Westat is a research company located n Rockville Maryland Westat * conducting this survey under contract to 
the U S Department of Education If you have any questions about the survey, contact Westat toll-free at 1-888- 
627-5304 


U.S. Department of Education 

National Center for Education Statistics 


OMB No 1850-0603 0M8 APPROVED 



The National Center tor Education Statutes is authorized to conduct ths survey under Section 9543. 20 US Code Your 
participation is voluntary Your answers may be used only for statistical purposes and may not be dsetosed, or used, m 
identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 
Pub*c Law 107-279. Section 183| The information you provide wd be combined with ^formation from other 
participants to produce statistical summaries and reports 
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Adult Training and Education Survey 


Start Here 


The Department at Education a studying adult 
framing and oducaton Each household If 
dfiereni and »e amk! your response so a* can 
contact you to admmttor a survey that rs nght 
for your household 

a Return tt»s tone won d no one \n your 
household is currently m an adult tramng or 
educahon program 

a This survey sftoud ho Mod out fry an adit 
household member Snog at the address 

a fhaose uaa a blue or mack pen it avatatlo 


1 How many p*opl* HI Mpf Qtt>- kM in 

this household? 

tactodeedUTs age 1 B and older who are 
temporarily many from Homo (tor euample 
attendmg coMage or imtvertdy) / they have 
no other permanent home 


m 


You (Adult 1) 

▼ 


2* What * hit or hat first name 
wibals. or nickname?. 

f m emmet m 0 tot 


S. Hom old Is this person In 

yM«i? .................. 


number age 10 or older 


► Continue answering questions 2 through 7 
tor each person age 10 or older krmg tn 
this household Start with yourself . 



4. What to thr» persons 

MI? 


ED*-. 

□ Mato 

□ female 


S. What »• the highest level or 
degree of schooling this 

person has completed? □ Lee* than fogh school 


MartKONEony 


□ Hgh school or GEO 

□ Some college no degree 

□ Two- year degree 

□ bachelor j Oegroo 

□ Hgher than bachelor's 


0. What to the best phone 
number to use to contact 
this person? 

fhi ton n ese etayhimi 


HUMir 


Wh*t typ. of phono rum bn* 
n Ihto? 


□ LonanowfnKl 

□ ColMdou 


Adult 2 

T 


cn 


m, 


□ Mato 

□ female 


□ Lass Wan hgh school 

□ High school or GCO 

□ Some c o hage. no degree 

□ Two year degree 

□ Bachelor's dogree 

□ Higher wen bachelor's degree 


□ Same As fidJi 1, or 


Adult 3 

▼ 


U.S. Department of Education 

NabonoJ Center for Education Stabs tics 


Adult 4 

▼ 


ff 


m, 


□ MMe 

□ female 


□ Lets than h*gh school 

□ Hrgh school or GED 

□ Some colege no degree 

□ Two year degree 

□ Bachelor's degree 

□ Hghar than bachelor's dsgree 


□ Same As Adult l or 


UO-HD-OID CED-dD-t 


p umnMorod 

□ CrtWifra 


□ LanSnoAMrad 

GCaWotrahno 


TTTT 

Tn 




m, 


□ Mbe 

□ female 


□ Lass than hgh school 

□ Hgh school or GCO 

□ Soma co ll ege no dogree 

□ Tw-o-yeer degree 

□ Bachelor's Oegroo 

□ Hgher than bechaior's 


□ Same As Adul 1 or 

cm-nn-t 


□ l 

□ CaShtoretoss 




► Pteasa verity that you have reported for each adult living in this household. If there are more than 4 adults bring In this household, pleats call 1 -088-427 -33 04 for instructions 


► Thank you. Pteasa return this form In the postage-peld envelope provided or mall It to: 
The Adult Training and Education Survey 
Wes tat 

1600 Research BNd RC BIS 
Rockville, MO 2*850-312? 


B-68 





U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CEN 


CATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


September 15.2010 


Dear |City| Household: 


Hie U.S. Department of Education is conducting an important national research study, and 
wc need your help. This short survey is the first step in a study about job and career training 
opportunities for people of all ages. It will let us know if anyone in your household is eligible 
for the study. 

Please fill out this short survey and mail it back in the postage-paid envelope that is provided. 

• It takes only a couple of minutes to fill out. 

• Every type of household in America needs to lie included. 

• If no one in your household is eligible, wc will not contact you again about this study. 

• The results of the study will help policymakers, researchers, and educators understand the 
job and career training needs of our diverse population in changing times. 

Your household was chosen as part of this voluntary scientific survey, and wc cannot replace you 
with someone else. All information you prov ide may only be used for statistical purposes and 
may not be disclosed, or used, in identifiable form for any other purpose except as required by 
law (Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279. Section 183]. 

Wc have enclosed a small cash token of our appreciation. 

Westat, a survey research group, is conducting this survey on our behalf. If you have questions 
or need assistance, please contact them at 1-888-627-5304. You can find more information about 
this study online at nces.cd.gov/ATES. 

Thank you for taking part in this important study. 


Sincerely. 



Stuart Kcrachsky ' — ' 

Acting Commissioner 

National Center for Education Statistics 


Enclosure 


WASHINGTON. D.C. 20006 
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U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CEN 


CATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


September 15, 2010 


Dear [City] Household: 

Pie U.S. Depart me til of Education is conducting an important national research study, and 
we need your help. This short survey is the first step in a study about job and career training 
opportunities for people of all ages. It will let us know if anyone in your household is eligible 
for the study. 

Please till out this short survey and mail it back in the postage-paid envelope that is provided. 

• it takes only a couple of minutes to fill out. 

• Every type of household in America needs to be included. 

• If no one in your household is eligible, we will not contact you again about this study. 

• The results of the study will help policymakers, researchers, and educators understand the 
job and career training needs of our diverse population in changing times. 

Your household was chosen as part of this voluntary scientific survey, and we cannot replace you 
with someone else. All information you provide may only be used for statistical purposes and 
may not be disclosed, or used, in identifiable form for any other purpose except as required by 
law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279. Section 183]. 

We have enclosed a small cash token of our appreciation. If someone in your household is 
selected fora telephone follow-up interview, wc will send an additional $10 after the telephone 
interview is completed. 

Wcstat. a survey research group, is conducting this survey on our behalf. If you have questions 
or need assistance, please contact them at 1-888-627-5504. You can find more information about 
this study online at nces.ed.gov/ ATES. 

Thank you for taking part in this important study. 


Sincerely. 



Enclosure 


Acting Commissioner 

National Center for Education Statistics 


WASHINGTON. D.C. 20006 
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U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CEN 


CATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


September 15.2010 


Dear (City] Household: 

The U.S. Department of Education is conducting an important national research study, and 
we need your help, 'litis short survey is the first step in a study about job and career training 
opportunities for people of all ages. It will let us know if anyone in your household is eligible 
for the study. 

Please fill out this short survey and mail it back in the postage-paid envelope that is provided. 

• It takes only a couple of minutes to fill out. 

• Every type of household in America needs to be included. 

• If no one in your household is eligible, we will not contact you again about this study. 

• The results of the study will help policymakers, researchers, and educators understand the 
job and career training needs of our diverse population in changing times. 

Your household was chosen as part of this voluntary scientific survey, and we cannot replace you 
with someone else. All information you provide may only be used for statistical purposes and 
may not be disclosed, or used, in identifiable form for any other purpose except as required by 
law (Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279. Section 183]. 

We have enclosed a small cash token of our appreciation. If someone in your household is 
selected for a telephone follow-up interview, we will send an additional S20 after the telephone 
interview is completed. 

Westat. a survey research group, is conducting this survey on our behalf. If you have questions 
or need assistance, please contact them at 1-888-627-5304. You can find more information about 
this study online at nces.ed.gov/ATKS. 

Thank you for taking part in this important study. 


Sincerely. 



Stuart Kcrachsky O' 

Acting Commissioner 

National Center for Education Statistics 


Enclosure 


WASHINGTON. D.C. 20006 
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U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 

NATIONAL CENTER FOR EDUCATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


Dear [City] Household: 


October 6.2010 


We recently mailed you a short survey about an important national research study and have not heard 
back from you. This short survey is the first step in a study about job and career training opportunities 
for people of all ages. It will let us know if anyone in your household is eligible for the study. 

Please till exit this short survey and mail it back in the postage-paid envelope that is provided. If 
you have already mailed in your survey and this letter has reached you late, we apologize for the 
duplication, and thank you! 

• It takes only a couple of minutes to lill exit. 

• Every type of household in America needs to be included. 

• If no one in your household is eligible, we will not contact you again about this study. 

• The results of the study will help policymakers, researchers, and educators understand the job 
and career training needs of our diverse population in changing times. 

Your household was chosen as part of this voluntary scientific survey , and we cannot replace you 
with someone else. All information you provide may only be used for statistical purposes and may 
not be disclosed, or used, in identifiable form for any other purpose except as required by law 
[Education Sciences Reform Act of 2002 (ESRA 2002) Public l.aw 107-279, Section 183]. 

Someone in your household may also be selected for a telephone follow-up interview. 

Westat. a survey research group, is conducting this survey on our behalf. If you have questions or 
need assistance, please contact them at I -888-627-5304. You can find more information about this 
study online at nces.ed.gov/ATES. 

Thank you for taking part in this important study. 

Sincerely. 

Stuart Kcrachsky ^ 

Acting Commissioner 

Enclosure National Center for Education Statistics 


WASHINGTON. D.C. 20006 
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U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CEN 


CATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


October 6.2010 


Dear [City 1 Household: 

We recently mailed you a short survey about an important national research study and have not 
heard back from you . This short survey is the first step in a study about job and career training 
opportunities for people of all ages. It will let us know if anyone in your household is eligible for 
the study. 

Please fill out this short survey and mail it back in the postage-paid envelope that is provided. If 
you have already mailed in your survey and this letter has reached you late, we apologize for the 
duplication. and thank you! 

• It takes only a couple of minutes to fill out. 

• Every type of household in America needs to be included. 

• If no one in your household is eligible, we will not contact you again about this study. 

• The results of the study w ill help policymakers, researchers, and educators understand the job 
and career training needs of our diverse population in changing times. 

Your household was chosen as part of this voluntary scientific survey , and we cannot replace you 
with someone else. All information you provide may only be used for statistical purposes and may 
not be disclosed, or used, in identifiable form for any other purpose except as required by law 
| Education Sciences Reform Act of 2<X)2 (HSRA 2(X)2) Public Law 107-279, Section 183|. 

If someone in your household is selected for a telephone follow-up interview, we will send $10 
after tl*e telephone interview is completed. 

Westat. a survey research group, is conducting this survey on our behalf. If you have questions or 
need assistance, please contact them at 1-888-627-5304. You can find more information about this 
study online at nces.ed.gov/ATES. 

Thank you for taking part in this important study. 


Sincerely. 



Stuart Kerachsky 

Acting Commissioner 

National Center for Education Statistics 


Enclosure 


WASHINGTON. O.C. 20006 
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U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CEN 


CATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


October 6.2010 


Dear [City] Household: 

We recently mailed you a short survey about an important national research study and have not 
heard back from you. This short survey is the first step in a study about job and career training 
opportunities for people of all ages. It will let us know if anyone in your household is eligible for 
the study. 

Please till out this short survey and mail it back in the postage-paid envelope that is provided. If 
you have already mailed in your survey and this letter has reached you late, we apologize for the 
duplication, and thank you! 

• It takes only a couple of minutes to fill out. 

• Every type of household in America needs to be included. 

• If no one in your household is eligible, we will not contact you again about this study. 

• The results of the study will help policymakers, researchers, and educators understand the job 
and career training needs of our diverse population in changing times. 

Your household was chosen as part of this voluntary scientific survey, and we cannot replace you 
with someone else. All information you provide may only be used for statistical purposes and may 
not be disclosed, or used, in identifiable form for any other purpose except as required by law 
(Education Sciences Reform Act of 2(X)2 (ESRA 2tX)2) Public Law 107-279. Section 183). 

If someone in your household is selected for a telephone follow-up interv iew, we will send $20 
after tl»e telephone interview is completed. 

Westat. a survey research group, is conducting this survey on our behalf. If you have questions or 
need assistance, please contact them at 1-888-627-5304. You can find more information about this 
study online at nccsed.gov/ATES. 

Thank you for taking part in this important study. 


Sincerely. 



Stuart Kerachsky (— ' 

Acting Commissioner 

National Center for Education Statistics 


Enclosure 


WASHINGTON. D.C. 20006 
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U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 

NATIONAL CENTER FOR EDUCATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


Dear [City] Household: 


October 20.2010 


We have been trying to reach you about an important national research study. Unfortunately, we have 
not yet received your completed questionnaire. 

We have ntailed this third questionnaire using rush delivery because of the importance of your 
response. 

This short survey is the first step in a study about job and career training opportunities for people 
of all ages. Please fill out this short surv ey and mail it back in the postage-paid envelope that is 
provided. If you have already mailed in your survey and this letter has reached you late, thank you! 

• It takes only a couple of minutes to fill out. 

• Every type of household in America needs to be included. 

• If no one in your household is eligible, we will not contact you again about this study. 

• The results of the study will help policymakers, researchers, and educators understand the job 
and career training needs of our diverse population in changing times. 

All information you provide may only be used for statistical purposes and may not be disclosed, 
or used, in identifiable form for any other purpose except as requited by law [Education Sciences 
Reform Act of 2002 (KSRA 2002) Public Law 107-279. Section 183). ' 


Someone in your household may also be selected for a telephone follow-up interview. 


Westat. a survey research group, is conducting this survey on our behalf. If you have questions or 
need assistance, please contact them at 1-888-627-5304. You can find more information about this 
study online at nees.ed.gov/ATES. 


Thank you for taking part in this important study. 


Enclosure 


Sincerely. 



Stuart Kerachsky 
Acting Commissioner 
National Center for Education Statistics 


WASHINGTON. D.C. 20006 
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U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CEN 


CATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


October 20. 2010 


Dear [City 1 Household: 

We have been trying to reach you about an important national research study . Unfortunately , we 
have not yet received your completed questionnaire. 

We have mailed this third questionnaire using rush delivery because of the importance of your 
response. 

This short survey is the lirst step in a study about job and career training opportunities for people 
of all ages. Please (ill out this short survey and mail it back in the postage-paid envelope that is 
provided. If you have already mailed in your survey and this letter has reached you late, thank you! 

• It takes only a couple of minutes to fill out. 

• Every type of household in America needs to be included. 

• I f no one in your household is eligible, we will not contact you again about this study. 

• The results of the study will help policy makers, researchers, and educators unde island the job 
and career training needs of our diverse population in changing times. 

All information you provide may only be used for statistical purposes and may not be disclosed, 
or used, in identifiable form for any other purpose except as required by law [Education Sciences 
Reform Act of 2002 (ESRA 2002)'public I .aw 107-279. Section I83|. 

If someone in your household is selected for a telephone follow-up interv iew, we will send SI0 
alter the telephone interview is completed. 

Westat. a survey research group, is conducting this survey on our behalf. If you have questions or 
need assistance, please contact them at I -888-627-5304. You can find more information about this 
study online at necsed.gov/ATES. 

Thank you for taking part in this important study. 


Sincerely. 



Stuart Kerachsky 

Acting Commissioner 

National Center for Education Statistics 


Enclosure 


WASHINGTON. D.C. 20006 


B-76 




U.S. DEPARTMENT OF EDUCATION 

INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CEN 


CATION STATISTICS 


|CITY| RESIDENT 
| ADDRESS | 

|CITY|. (ST) |ZIPCODE)-|ZIP4] 


October 20. 2010 


Dear [City 1 Household: 

We have been trying to reach you about an important national research study. Unfortunately, we 
have not yet received your completed questionnaire. 

We have mailed this third questionnaire using rush delivery because of the importance of your 
response. 

This short survey is the first step in a study about job and career training opportunities for people 
of all ages. Please fill out this short survey and mail it back in the postage-paid envelope that is 
provided. If you have already mailed in your survey and this letter has reached you late, thank you! 

• It takes only a couple of minutes to fill out. 

• Every type of household in America needs to be included. 

• If no one in your household is eligible, we will not contact you again about this study. 

• The results of the study will help policymakers, researchers, and educators understand the job 
and career training needs of our diverse population in changing times. 

All information you provide may only be used for statistical purposes and may not be disclosed, 
or used, in identifiable form for any other purpose except as requited by law (Education Sciences 
Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section !83|. " 

If someone in your household is selected for a telephone follow-up interview, we will send S20 
after live telephone interview is completed. 

Westat. a survey research group, is conducting this survey on our behalf. If you have questions or 
need assistance, please contact them at I -888-627-5304. You can find more information about this 
study online at nces.ed.gov/ATES. 

Thank you for taking part in this important study. 


Sincerely. 



Stuart Kerachsky ' — 

Acting Commissioner 

National Center for Education Statistics 


Enclosure 


WASHINGTON. D.C. 20006 
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September 22.2010 

Last week we mailed you a questionnaire for the Adult Training and Education 
Survey, sponsored by the U.S. Department of Education. If someone in your 
household has already returned the survey, we thank you very much for your help. 

If you have not completed and returned your survey, please do so right away. We 
need to hear from everyone and your help is very important to us. Your household 
was randomly selected to take part in this survey and cannot be replaced. 

If you have any questions, feel free to contact us at 1 -868-627-5304. 


Sincerely. 



Stuart Kerachsky. Acting Commissioner, 
National Center for Education Statistics 
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Table C-1. Percentage distribution of respondents, by highest reported level of 
education: 2010-11 


Highest reported level of education 

Percent 

Total 

100.0 

Less than high school 

9.0 

Grades 1-11 

6.6 

12th grade, no diploma 

2.4 

High school (or equivalent) 

24.9 

Regular high school diploma 

22.3 

GED or alternative credential 

2.7 

Some college or associate's degree 

25.8 

Some college credit, but less than 1 year 

4.8 

1 or more years of college credit, no degree 

13.2 

Associate's degree 

7.8 

Bachelor's degree 

23.4 

Graduate or professional degree 

16.9 

Master's degree 

12.4 

Professional degree beyond a bachelor's degree 

1.9 

Doctorate degree 

2.5 


NOTE: Figures represent unweighted estimates. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 


Table C-2. Percentage of respondents reporting a certification/license or 


certificate: 2010-11 

Reporting of credentials 

Percent 

Certification(s)/license(s) 

33.1 

Certificate(s) 

14.1 

No credentials were reported 

58.8 


NOTE: Figures represent unweighted estimates. Respondents were able to select multiple credentials; therefore, column 
totals may not add to 100 percent. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 
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Table C-3. Percentage distribution of respondents who reported any credentials, percentage of respondents who 
reported any certification/license or certificate, and percentage distribution of certification/license or 
certificate among those who reported any credentials, by respondent characteristics: 2010-11 


Percentage distribution of Percentage of respondents Among those who reported credentials, percentage 

respondents who reported who reported distribution of respondents who reported 


Respondent characteristics 

No credentials 

Any 

credentials 

Any 

certification/ 

license 

Any 

certificate 

Certification/ 
license only 

Certificate 

only 

Both a 

certification/license 
and certificate 

Total 

58.8 

41.2 

33.1 

14.1 

65.7 

19.7 

14.6 

Sex 

Male 

59.3 

40.7 

33.0 

13.6 

66.6 

18.8 

14.6 

Female 

58.4 

41.6 

33.2 

14.5 

65.1 

20.3 

14.7 

Race/ethnicity 1 

White 

58.8 

41.3 

33.9 

13.4 

67.6 

17.8 

14.7 

Black 

58.4 

41.6 

29.8 

17.9 

56.9 

28.5 

14.6 

Hispanic 

58.5 

41.5 

33.9 

15.3 

63.3 

18.4 

18.4 

Asian 

64.1 

35.9 

28.2 

10.7 

70.3 

21.6 

8.1 

Other 

61.8 

38.2 

23.5 

17.7 

53.9 

38.5 

7.7 

Two or more races 

55.8 

44.2 

27.9 

22.1 

50.0 

37.0 

13.0 

Age 

18 to 24 years old 

83.8 

16.2 

14.9 

3.1 

81.1 

8.1 

10.8 

25 to 34 years old 

56.2 

43.8 

37.8 

11.0 

74.9 

13.7 

11.4 

35 to 44 years old 

53.0 

47.0 

41.2 

12.9 

72.5 

12.4 

15.1 

45 to 54 years old 

53.0 

47.0 

37.4 

18.3 

61.0 

20.5 

18.5 

55 to 64 years old 

53.4 

46.6 

35.6 

17.6 

62.3 

23.6 

14.1 

65 to 74 years old 

57.8 

42.2 

32.9 

15.9 

62.3 

22.1 

15.6 

75 years old and over 

74.3 

25.7 

17.9 

10.6 

59.0 

30.3 

10.7 


See notes at end of table. 
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Table C-3. Percentage distribution of respondents who reported any credentials, percentage of respondents who 
reported any certification/license or certificate, and percentage distribution of certification/license or 
certificate among those who reported any credentials, by respondent characteristics: 2010-11 — 
Continued 



Percentage distribution of 
respondents who reported 

Percentage of respondents 
who reported 

Among those who reported credentials, percentage 
distribution of respondents who reported 

Respondent characteristics 

No credentials 

Any 

credentials 

Any 

certification/ Any 

license certificate 

Certification/ 
license only 

Certificate 

only 

Both a 

certification/license 
and certificate 

Highest reported level of 
education 

Less than high school 

78.6 

21.4 

14.0 

10.4 

51.4 

24.7 

13.9 

Grades 1-11 

82.2 

17.8 

12.2 

8.9 

50.0 

31.8 

18.2 

12th grade, no diploma 

68.9 

31.1 

18.9 

14.4 

53.6 

39.3 

7.1 

High school (or equivalent) 

72.1 

27.9 

19.7 

12.8 

54.1 

29.3 

16.6 

Regular high school 
diploma 

71.8 

28.2 

20.0 

13.0 

53.9 

29.1 

17.1 

GED or alternative 
credential 

74.8 

25.3 

17.2 

11.1 

56.0 

32.0 

12.0 

Some college or 
associate’s degree 

54.6 

45.4 

31.6 

23.3 

48.7 

30.4 

20.8 

Some college credit, but 
less than 1 year 

63.1 

36.9 

21.8 

23.5 

36.4 

40.9 

22.7 

1 or more years of college 
credit, no degree 

54.3 

45.8 

32.2 

22.7 

50.4 

29.7 

19.9 

Associate's degree 

49.8 

50.2 

36.7 

24.2 

51.7 

26.9 

21.4 

Bachelor's degree 

54.9 

45.1 

40.1 

9.8 

78.4 

10.9 

10.7 

Graduate or professional 
degree 

40.3 

59.7 

55.7 

10.2 

83.0 

6.7 

10.4 

Master's degree 

42.0 

58.0 

53.2 

11.2 

80.7 

8.2 

11.2 

Professional degree beyond 
a bachelor's degree 

20.8 

79.2 

76.4 

11.1 

86.0 

3.5 

10.5 

Doctorate degree 

46.8 

53.2 

52.1 

4.3 

92.0 

2.0 

6.0 


See notes at end of table. 
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Table C-3. Percentage distribution of respondents who reported any credentials, percentage of respondents who 
reported any certification/license or certificate, and percentage distribution of certification/license or 
certificate among those who reported any credentials, by respondent characteristics: 2010-11 — 
Continued 

Among those who reported credentials, 

Percentage distribution of Percentage of respondents percentage distribution of respondents who 

respondents who reported who reported reported 

Any Both a 




Any 

certification/ 

Any 

Certification/ 

Certificate 

certification/license 

Respondent characteristics 

No credentials 

credentials 

license 

certificate 

license only 

only 

and certificate 

Labor force status 

Employed 

52.0 

48.0 

41.4 

13.6 

71.7 

13.9 

14.4 

Unemployed (seeking 

employment) 

64.3 

35.7 

25.9 

16.5 

53.7 

27.4 

19.0 

Not in labor force 

67.8 

32.2 

22.3 

14.2 

55.8 

30.8 

13.4 

Labor force status unknown 

47.5 

52.5 

47.5 

22.5 

57.1 

9.5 

33.3 

Immigration status 

Born in U.S. 

59.0 

41.0 

32.9 

14.1 

65.5 

19.8 

14.7 

Born outside of U.S. 

56.4 

43.6 

35.6 

14.2 

67.4 

18.4 

14.3 


1 Black includes African American, Other includes Pacific Islanders and American Indians (including Alaska Natives), and two or more races includes non-Hispanics reporting 
more than one race. Race categories exclude persons of Hispanic ethnicity. 

NOTE: Figures represent unweighted estimates. Imputed data were used for all respondent characteristics except labor force status, for which imputed data were not available. 
Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. 
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Table C-4. Percentage distribution of respondents reporting certifications/licenses, 
by number of certifications/licenses reported: 2010-11 


Number of certifications/licenses reported 

Percent 

Total 

100.0 

None 

66.5 

One or more 

33.0 

1 

21.3 

2 

6.9 

3 

2.6 

4 

0.9 

5 or more 

0.9 

Number not specified 

# 

Don't know 

0.5 

# Rounds to zero. 

NOTE: Figures represent unweighted estimates. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

Table C-5. Percentage distribution of certifications/licenses, by respondents’ 
classification of certification/license type: 2010-11 

Respondents' classification of certification/license type 

Percent 

Total 

100.0 

Certification only 

44.9 

License only 

29.4 

Certification and license both 

22.0 

Data missing or not collected 

3.7 

Don't know 

3.0 

Refused 

# 

Not collected 

# 


# Rounds to zero. 

NOTE: Figures represent unweighted estimates. Survey question was asked for the first five certifications and licenses 
identified per respondent for 1,196 respondents. Includes self-reported data only; data completed by proxy were excluded 
from this analysis. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 
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Table C-6. Percentage distribution of respondents’ five most recent certifications/ 
licenses and percentage distribution of respondents’ most recent 
certification/license, by Standard Occupational Classification (SOC): 
2010-11 


Standard Occupational Classification (SOC) 

Percentage distribution of recently earned 
certifications/licenses 

Five most recent Most recent 

Total 

100.0 

100.0 

Architecture, engineering, and information technology 

5.6 

4.5 

Computer occupations 

3.1 

2.2 

Architects, surveyors, cartographers and engineers 

2.0 

2.0 

Drafters, engineering technicians, and mapping technicians 

# 

# 

Media and communication equipment workers 

# 

# 

Business 

15.5 

16.4 

Management occupations 

3.9 

4.3 

Business and financial operations 

2.7 

3.3 

Financial and insurance sales occupations 

4.4 

3.3 

Real estate brokers and sales agents 

2.7 

3.3 

Other sales and related occupations 

0.8 

0.9 

Office and administrative support occupations 

1.0 

1.3 

Health 

23.3 

24.7 

Health diagnosing and treating practitioners, except nurses 

3.7 

4.4 

Health technologists and technicians 

6.8 

6.7 

Nursing occupations 

7.1 

7.4 

Health care support occupations 

5.7 

6.2 

Personal services 

4.2 

4.8 

Personal appearance workers (cosmetologists, etc.) 

1.8 

2.3 

Other personal care and service workers 

2.4 

2.6 

Public and social services 

10.1 

10.2 

Community and social service occupations 

4.0 

4.3 

Lawyers, judges, and related workers 

2.9 

2.8 

Legal support workers 

# 

# 

Protective service occupations 

2.8 

2.7 


See notes at end of table. 
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Table C-6. Percentage distribution of respondents’ five most recent certifications/ 
licenses and percentage distribution of respondents’ most recent 
certification/license, by Standard Occupational Classification (SOC): 
201 0-1 1 — Continued 


Standard Occupational Classification (SOC) 

Percentage distribution of recently earned 
certifications/licenses 

Five most recent Most recent 

Education 

20.4 

19.6 

Pre-K-12 teachers 

18.2 

18.0 

Other education and training occupations 

2.3 

1.7 

Trades 

14.8 

15.0 

Construction and extraction occupations 

3.8 

3.6 

Vehicle and mobile equipment mechanics, installers, and repairers 

1.8 

1.3 

Heating, air conditioning, refrigeration mechanics, and installers 

0.8 

0.7 

Other installation, maintenance, and repair workers 

1.4 

1.4 

Production occupations, except food production 

1.6 

1.8 

Transportation and material moving occupations 

5.4 

6.3 

Other 

4.0 

4.3 

Mathematics and science occupations 

0.6 

0.5 

Art and design workers 

# 

# 

Entertainers and performers, sports, and related workers 

# 

# 

Media and communication workers 

# 

# 

Food processing and preparation occupations 

1.3 

1.5 

Building cleaning and grounds maintenance occupations 

0.9 

1.0 

Farming, fishing, and forestry occupations 

# 

# 

Missing or cannot determine placement 

2.3 

0.6 


# Rounds to zero. 

NOTE: Figures represent unweighted estimates. Includes self-reported data only; data completed by proxy were 

excluded from this analysis. Detail may not sum to totals because of rounding. Respondents were asked to report the credential 

most recently earned, followed by the next most recently earned and so on up to five credentials. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education 
Survey (ATES) Pilot Study, 2010-11. 
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Table C-7. Percentage distribution of most recent certification/license, by number 
of weeks needed to complete certification/license: 2010-11 

Number of weeks needed to earn most recent certification/license Percent 


Total 

100.0 

Earned through a degree program 

22.0 

Less than 1 week 

6.0 

1 to 4 weeks 

12.5 

5 to 12 weeks 

14.1 

1 3 to 24 weeks 

7.9 

25 to 52 weeks 

10.0 

53 to 104 weeks 

8.9 

Greater than 104 weeks 

12.9 

Data missing or not collected 

5.9 


NOTE: Figures represent unweighted estimates. Survey questions were asked only for the most recent work-related 
certification/license or, if none were identified as work-related, for the most recent certification/license. The average 
number of weeks taken to earn the most recent certification/license was 54 (n = 863). Includes self-reported data only; 
data completed by proxy were excluded from this analysis. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

Table C-8. Percentage distribution of certifications/licenses related to respondents’ 
current jobs, by certification/license requirement: 2010-11 


Certification/license requirement 

Percent 

Total 

100.0 

Required for current job 

72.8 

Not required for current job 

27.2 


NOTE: Figures represent unweighted estimates. Survey questions were asked only for the most recent work-related 
certification/license or, if none were identified as work-related, for the most recent certification/license. Additionally, this 
survey question was only asked of those respondents who reported being currently employed. Includes self-reported data 
only; data completed by proxy were excluded from this analysis. Detail may not sum to totals because of rounding. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 
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Table C-9. Percentage of respondents’ self-reported benefit from earning most 

recent certification/license or certificate, by benefit type and number of 


benefits: 2010-11 

Benefit type and number of benefits Percent 


Benefit type 

Promotion 2.2 

Higher pay or bonus 22.2 

Career advancement 12.2 

Is just mandatory or required for job 1 9.3 

Improved job performance 5.2 

Helped me stay current with new regulations, laws, or technologies 6.3 

To change job or career field, enter the workforce, or start own business 18.2 

No benefits received 10.2 

Other 32.3 

Don't know/Refused 0.8 

Number of benefits 

1 76.2 

2 18.2 

3 4.0 

4 0.8 

Missing 0.9 


NOTE: Figures represent unweighted estimates. Includes self-reported data only; data completed by proxy were 
excluded from this analysis. Question was asked as an open-ended question. Interviewers were instructed to select all 
that apply; therefore, column totals may exceed 100 percent. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

Table C-10. Percentage distribution of respondents reporting certificates, by number 
of certificates reported: 2010-11 


Number of certificates reported Percent 

Total MO 

None 85.4 

One or more 14.1 

1 9.9 

2 2.1 

3 1.0 

4 # 

5 or more 0.6 

Number not specified # 

Don't know/Refused 0.5 

# Rounds to zero. 


NOTE: Figures represent unweighted estimates. Detail may not sum to totals because of rounding 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 

(ATES) Pilot Study, 2010-11. 
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Table C-11. Percentage distribution of respondents’ five most recent certificates and 
most recent certificate, by Classification of Instructional Program (CIP): 
2010-11 


Classification of Instructional Program 

Percentage distribution of recently earned 
certificates 

Five most recent Most recent 

Total 

100.0 

100.0 

Architecture, engineering, and information technology 

14.3 

14.5 

Architecture and engineering 

3.1 

3.9 

Communications technologies/technologists 

0.6 

0.6 

Computer and information sciences 

7.9 

6.8 

Engineering and related technologies 

2.7 

3.3 

Business 

13.6 

14.5 

Business management 

6.5 

6.8 

Business support 

4.5 

5.8 

Marketing 

2.7 

1.9 

Health 

15.3 

18.6 

Health professions, except nursing 

1.2 

1.6 

Nursing 

3.9 

5.6 

Health technologists and technicians 

6.6 

7.8 

Health aides 

3.6 

3.7 

Personal services 

7.1 

8.9 

Cosmetology 

3.7 

5.2 

Culinary arts 

1.9 

2.1 

Personal services (other than cosmetology and culinary arts) 

1.5 

1.6 

Public and social services 

8.9 

8.7 

Legal professions 

# 

# 

Legal support services 

1.2 

1.6 

Library sciences 

# 

# 

Protective services 

5.1 

3.5 

Public administration and human services 

2.2 

3.1 


See notes at end of table. 
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Table C-11. Percentage distribution of respondents’ five most recent certificates and 
most recent certificate, by Classification of Instructional Program (CIP): 
201 0-1 1 — Continued 


Classification of Instructional Program 

Percentage distribution of recently earned 
certificates 

Five most recent Most recent 

Education 

5.0 

4.7 

Education 

5.0 

4.7 

Trades 

20.4 

18.6 

Construction trades 

5.6 

5.6 

Manufacturing 

2.7 

2.7 

Mechanic and repair technologies 

8.5 

6.2 

Transportation and material moving 

3.6 

4.1 

Other 

5.9 

5.6 

Visual and performing arts 

1.7 

1.2 

Humanities and general studies 

0.5 

0.8 

Mathematics and science 

0.6 

0.6 

Psychology 

# 

# 

Social sciences and history 

# 

# 

Agriculture and natural resources 

1.8 

1.9 

Communications, journalism, and related programs 

0.8 

0.6 

Missing or cannot determine placement 

9.7 

5.8 


# Rounds to zero. 

NOTE: Figures represent unweighted estimates. Includes self-reported data only; data completed by proxy were excluded from 
this analysis. Detail may not sum to totals because of rounding. Respondents were asked to report the credential most recently 
earned, followed by the next most recently earned and so on up to five credentials. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) 
Pilot Study, 2010-11. 
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Table C-12. Percentage distribution of respondents’ certificate completion status: 
2010-11 ~ 


Certificate completion status 

Percent 

Total 

100.0 

Certificate completed 

93.3 

Certificate not completed 

1.8 

Don't know 

2.2 

Refused 

0.9 

Missing/Not collected 

1.7 


NOTE: Figures represent unweighted estimates. Survey question was asked for the first five certificates identified per 
respondent for 520 respondents. Includes self-reported data only; data completed by proxy were excluded from this 
analysis. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 
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Table C-13. Percentage distribution of most recent certificates, by number of credit 
hours or hours required to complete the certificate program: 2010-11 


Number of credit hours or hours 

Percent 

Reported in credit hours 

100.0 

1 to 4 credit hours 

14.6 

5 to 8 credit hours 

6.1 

9 to 12 credit hours 

6.1 

13 to 20 credit hours 

15.9 

21 or more credit hours 

57.3 

Reported in hours 

100.0 

1 to 4 actual hours 

2.8 

5 to 8 actual hours 

5.6 

9 to 12 actual hours 

3.5 

13 to 20 actual hours 

6.9 

21 to 50 actual hours 

23.6 

51 to 75 actual hours 

6.9 

76 to 100 actual hours 

6.9 

101 to 200 actual hours 

16.7 

201 to 300 actual hours 

2.1 

301 to 400 actual hours 

2.1 

401 to 500 actual hours 

2.1 

501 or more actual hours 

20.8 

Data missing or not collected 

100.0 

Don't know 

93.8 

Refused 

1.0 

Missing 

5.2 


NOTE: Figures represent unweighted estimates. Survey questions were asked only for the most recent certificate. The 
average number of credit hours needed to earn the most recent certificate was 92, and the average number of actual 
hours needed to earn the most recent certificate was 388. When credit hour responses were capped at 150, the average 
number of credit hours needed was 32 and when actual hour responses were capped at 2,100, the average number of 
actual hours needed was 334. Includes self-reported data only; data completed by proxy were excluded from this 
analysis. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 
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Table C-14. Percentage of respondents’ benefits received by earning most recent 
certificate that was not part of the coursework or training taken for a 
certification or license, by benefit type and number of benefits: 
2010-11 


Benefit type and number of benefits 

Percent 

Type of benefit 

Promotion 

2.0 

Higher pay or bonus 

7.8 

Career advancement 

10.6 

Is just mandatory or required for job 

9.0 

Improved job performance 

5.9 

Helped me stay current with new regulations, laws, or technologies 

6.7 

To change job or career field, enter the workforce, or start own business 

20.5 

No benefits received 

13.2 

Other 

37.0 

Don't know/Refused/Missing 

3.4 

Number of benefits 

1 

84.0 

2 

9.8 

3 or more 

2.8 

Missing 

3.4 


NOTE: Figures represent unweighted estimates. Survey question was asked only for the most recent certificate. Includes 
self-reported data only; data completed by proxy were excluded from this analysis. Question was asked as an open- 
ended question. Interviewers were instructed to select all that apply; therefore, column totals may exceed 100 percent. 
Additionally, questions asked about the most recent certification and licensure were not asked again about the most 
recent certificate if a respondent said that their most recent certificate was part of the coursework or training taken for “a 
certification or license” at item CT2C. However, because the ATES pilot interviews did not explicitly confirm that the most 
recent certificate was part of the coursework or training taken for the most recent certification or license, the direct reports 
(those reported directly in the certificate section) are presented separately from the indirect reports (those who said “Yes” 
to item CT2C and therefore reported in the certification/license section). Detail may not sum to totals because of 
rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 
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Table C-15. Percentage of certificates, by whether they were identified to be a part of 
the coursework or training taken for a previously reported 


certification/license: 2010-11 

Certificate holder reporting of certifications/licenses Percent 

Certification/license was reported 43.1 

Certificate is part of coursework or training taken for certification/license 71.1 

Certificate is not part of coursework or training taken for certification/license 21 .8 


NOTE: Figures represent unweighted estimates. Includes respondents' first five reported certificates only. Includes self- 
reported data only; data completed by proxy were excluded from this analysis. Question was asked only if respondent 
previously reported having a certification/license. In all, the question was asked of 43.1 percent of all reported certificates 
where the respondent also reported having a certification/license. Detail may not sum to totals because of rounding. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 

Table C-16. Percentage distribution of respondents, by response to the 

certification/license question and interviewers' use of the certification 


probe: 2010-11 




Percent of 

Percent of respondents 

Interviewers' 



respondents not 

answering “Don't 

use of the 


Percent of 

reporting a 

know” or “Refused” to 

certification 


respondents reporting 

certification/ 

certification/ license 

probe 

Total percent 

a certification/license 

license 

question 

Probe used 


33.6 

64.5 

1.9 

Probe not used 

100.0 

32.8 

67.0 

# 

# Rounds to zero. 


NOTE: Figures represent unweighted estimates. The probe was used with 17 percent of all respondents. Includes self- 
reported data only; data completed by proxy were excluded from this analysis. Detail may not sum to totals because of 
rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 


Table C-17. Percentage distribution of respondents, by response to the certificate 
question and interviewers' use of the certificate probe: 2010-11 


Interviewers' 
use of the 
certificate probe 

Total percent 

Percent of 
respondents reporting 
a certificate 

Percent of 
respondents not 
reporting a 
certificate 

Percent of respondents 
answering “Don't 
know” or “Refused” to 
certificate 

Probe used 

100.0 

26.6 

71.9 

1.5 

Probe not used 

100.0 

12.1 

87.7 

# 


# Rounds to zero. 

NOTE: Figures represent unweighted estimates. The probe was used with 15 percent of all respondents. Includes self- 
reported data only; data completed by proxy were excluded from this analysis. Detail may not sum to totals because of 
rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES) Pilot Study, 2010-11. 
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Table C-18. Percentage distribution of respondents in seeded sample reporting 

certification/license not matching certification/license reported in seeded 
certification/license frame, by credential type: 2010-11 


Credential type 1 

Percent of 
certifications 

Percent of licenses 

CN4. “Is that a certification, a license, or both?” 

Certification 

53.8 

22.1 

License 

31.2 

50.8 

Both 

10.8 

23.0 

Don't know/Refused/Missing 

4.3 

4.1 

CN6. “Did you get this certification/license mainly for work-related 
reasons or mainly for personal interest?” 

Mainly work-related 

85.0 

91.8 

Mainly personal interest 

11.8 

4.9 

Don't know/Refused/Missing 

3.2 

3.3 


1 1tems asked of the first five certifications or licenses, or both. 

NOTE: Figures represent unweighted estimates. Includes self-reported data only; data completed by proxy were excluded 
from this analysis. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey 
(ATES), Pilot Study, 2010-11. 
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Table C-19. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and selected survey items, with “don’t know” and “refused” set to 
missing: 2010-11 


Certification/licensure item 

Self- 

reported 

estimate 

Proxy-reported 

estimate 

Percentage 

difference 1 

CN1. Now I'd like to ask you about professional certification and 
licensure. 

Do you/person have a professional certification or a state or industry 
license? 

Yes 

33.1 

32.6 

-0.5 

No 

66.9 

67.4 

0.5 

CN1A. Have more than one certification or license? 

Yes 

35.0 

32.0 

-3.0 

No 

65.0 

68.0 

3.0 

CN1B. How many certifications do you/person have? 

1 

65.2 

71.3 

6.1 

2 

21.3 

19.6 

-1.7 

3 

7.9 

6.3 

-1.6 

4 

2.9 

1.7 

-1.2 

5 or more 

2.7 

1.0 

-1.7 

CN3AR1. Name of most recent certification 

Name reported 

100.0 

99.0 

-1.1 

CN4R1. Is that a certification, a license, or both? 

Certification 

42.3 

37.4 

-5.0 

License 

43.5 

32.0 

-11.4 

Both 

23.2 

30.6 

7.4 

CN6R1. Did you/person get this certification mainly for work-related 
reasons, or mainly for personal interest? 

Mainly work-related 

92.0 

92.4 

0.4 

Mainly personal interest 

8.0 

7.6 

-0.4 


See notes at end of table. 
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Table C-19. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and selected survey items, with “don’t know” and “refused” set to 
missing: 2010-11 — Continued 


Certification/licensure item 

Self-reported 

estimate 

Proxy- 

reported 

estimate 

Percentage 

difference 1 

CN7A. Did you/person take the coursework or training for the certification 
as part of a college or university certificate or degree program? 

Yes 

55.9 

60.4 

4.5 

No 

44.1 

39.6 

-4.5 

CN7B. Was that a certificate program, an associate's degree program, a 
bachelor's degree program, or something else? 

Certificate 

18.8 

11.0 

-7.7 

Associate's 

10.7 

11.7 

1.0 

Bachelor's 

34.6 

42.9 

8.3 

Other 

36.0 

34.4 

-1.6 


CN7C. Was the coursework or training mainly self-study or mainly 


classes 

or courses with an instructor? 


Mainly self-study 

11.9 

11.4 

-0.5 

Mainly with an instructor 

88.1 

88.7 

0.5 

CN8NR. How many weeks was the course? 

Less than 1 week 

8.3 

8.6 

0.2 

1 to 4 weeks 

17.4 

16.5 

-0.9 

5 to 12 weeks 

19.5 

19.7 

0.3 

1 3 to 24 weeks 

10.9 

7.2 

-3.7 

25 to 52 weeks 

13.8 

11.2 

-2.6 

53 to 104 weeks 

12.3 

19.1 

6.8 

Greater than 104 weeks 

17.8 

17.8 

-0.1 

CN9NR. How many hours per week with an instructor? 

Less than 1 hour 

3.8 

3.8 

0.0 

1 to 4 hours 

14.0 

9.5 

-4.5 

5 to 8 hours 

19.4 

20.0 

0.6 

9 to 12 hours 

10.2 

8.6 

-1.6 

13 to 20 hours 

17.1 

17.1 

0.1 

21 to 50 hours 

34.4 

38.1 

3.7 

51 or more hours 

1.1 

2.9 

1.7 

CN10A. Did you/person have to do any of the following to get this — 
demonstrate skills while on the job? 

Yes 

66.2 

72.8 

6.7 

No 

33.8 

27.2 

-6.7 

CN10B — pass a test or exam? 

Yes 

91.1 

94.1 

3.0 

No 

8.9 

5.9 

-3.0 


See notes at end of table. 
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Table C-19. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and selected survey items, with “don’t know” and “refused” set to 
missing: 2010-11 — Continued 


Certification/licensure item 

Self- 

reported 

estimate 

Proxy- 

reported 

estimate 

Percentage 

difference 1 

CN10C. Submit a portfolio of work? 

Yes 

32.3 

33.2 

0.9 

No 

CN1 1A. Have to take. ..continuing education classes or earn CEUs to 
maintain it? 

67.7 

66.8 

-0.9 

Yes 

64.3 

63.6 

-0.8 

No 

35.7 

36.4 

0.8 

CN1 1 B — periodic tests? 

Yes 

31.6 

37.1 

5.5 

No 

68.4 

62.9 

-5.5 

CN12. Is this certification related to current job? 

Yes 

82.0 

85.6 

3.6 

No 

18.4 

14.4 

-3.6 

CN13. Is it required for current job? 

Yes 

72.8 

82.5 

9.7 

No 

CN14. Issuing organization 

27.2 

17.5 

-9.7 

CN14. [certified] [licensed] by state 

Yes 

68.9 

76.3 

7.4 

No 

31.1 

23.7 

-7.4 

CN14. [certified] [licensed] by industry 

Yes 

2.6 

4.2 

1.6 

No 

97.4 

95.8 

-1.6 

CN14. [certified] [licensed] by company 

Yes 

5.2 

2.8 

-2.4 

No 

94.8 

97.2 

2.4 


See notes at end of table. 
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Table C-19. Percentage distribution of responses to certification/license survey 

items, by reporting method, difference between the reporting method, 
and selected survey items, with “don’t know” and “refused” set to 
missing: 2010-11 — Continued 

Self- Proxy- 

reported reported Percentage 

Certification/licensure item estimate estimate difference 1 

CN14. Issuing Organization-Continued 
CN14. [certified] [licensed] by professional association 


Yes 

16.1 

11.5 

-4.6 

No 

83.9 

88.5 

4.6 

CN14. [certified] [licensed] by other body 

Yes 

14.0 

10.1 

-3.9 

No 

86.0 

89.9 

3.9 

CN15A. Can this certification be... revoked or suspended for any reason? 

Yes 

83.8 

83.5 

-0.3 

No 

16.3 

16.5 

0.3 

CN15B — used if you/person wanted to get a job with any employer in that 
field? 

Yes 

95.5 

98.0 

2.5 

No 

4.5 

2.0 

-2.5 

CN17. Is the certification currently valid or has it expired? 

Currently valid 

74.6 

71.2 

-3.3 

Expired 

25.5 

28.8 

3.3 

CN18. Do you/person plan to renew it? 

Yes 

11.5 

t 

t 

No 

88.5 

t 

T 


t Not applicable. 

t Data suppressed because reporting standards were not met and estimates are unreliable. 

1 Difference is the proxy percentage minus the self-reported percentage. 

NOTE: Figures represent unweighted estimates. Only valid responses are reported in the table; respondents not asked 
questions because of legitimate skips due to earlier responses are excluded. Proxy reports are when the respondent to the main 
interview answered questions about another randomly selected adult in the household. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES), 
Pilot Study, 2010-11. 
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Table D-1 . Standard errors for Table 2-3: Number and percentage distribution of respondents who reported any 

credentials, percentage of adults who reported any certification/license or certificate, and among those 
who reported any credentials, percentage distribution who reported a certification/license only, a 
certificate only, or both credentials, by respondents characteristics: 2010-11 




Percentage distribution of 
respondents who reported 

Percentage of respondents 
who reported 

Among those who reported any credentials, 
percentage distribution of respondents who 
reported 





Any 




Both a 
certification/ 

Respondent characteristics 

Number (in 
thousands) 

No 

credentials 

Any 

credentials 

certification/ 

license 

Any 

certificate 

Certification/ 
license only 

Certificate 

only 

license and 
certificate 

Total 

# 

0.90 

0.90 

0.84 

0.66 

1.59 

1.16 

1.25 

Sex 

Male 

# 

1.40 

1.40 

1.27 

0.99 

2.36 

1.47 

1.92 

Female 

# 

1.27 

1.27 

1.12 

0.92 

2.05 

1.73 

1.49 

Race/ethnicity 1 

White 

# 

1.05 

1.05 

0.97 

0.82 

1.77 

1.4 

1.27 

Black 

# 

2.93 

2.93 

2.73 

2.47 

5.49 

4.44 

3.44 

Hispanic 

# 

3.36 

3.36 

3.24 

2.2 

6.16 

5.02 

4.77 

Asian 

# 

4.81 

4.81 

4.61 

3.30 

9.38 

6.15 

8.53 

Other 

# 

9.66 

9.66 

9.97 

5.60 

15.95 

15.23 

5.94 

Two or more races 

# 

6.4 

6.34 

5.12 

4.70 

9.03 

8.67 

6.60 

Age 

1 8 to 24 years old 

1,203.5 

2.36 

2.36 

2.38 

1.00 

7.57 

5.56 

6.01 

25 to 34 years old 

1,760.0 

2.71 

2.71 

2.43 

1.48 

3.17 

2.97 

2.50 

35 to 44 years old 

1,439.2 

2.45 

2.45 

2.33 

1.71 

3.74 

2.53 

3.00 

45 to 54 years old 

1,399.4 

2.29 

2.29 

2.19 

1.75 

3.00 

3.02 

2.68 

55 to 64 years old 

1,085.0 

2.12 

2.12 

2.17 

1.50 

3.01 

2.92 

2.35 

65 to 74 years old 

894.3 

2.23 

2.23 

1.91 

1.87 

3.84 

3.56 

2.68 

75 years old and over 

828.8 

2.42 

2.42 

1.96 

1.79 

6.22 

4.73 

4.42 


See notes at end of table. 
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Table D-1 . Standard errors for Table 2-3: Number and percentage distribution of respondents who reported any 

credentials, percentage of adults who reported any certification/license or certificate, and among those 
who reported any credentials, percentage distribution who reported a certification/license only, a 
certificate only, or both credentials, by respondents characteristics: 2010-11 — Continued 




Percentage distribution of 
respondents who reported 

Percentage of 
respondents who reported 

Among those who reported any credentials, 
percentage distribution of respondents who 
reported 





Any 




Both a 
certification/ 

Respondent characteristics 

Number (in 
thousands) 

No 

credentials 

Any 

credentials 

certification/ Any 

license certificate 

Certification/ 
license only 

Certificate 

only 

license and 
certificate 

Highest reported level of education 









Less than high school 

# 

2.43 

2.43 

2.37 

1.65 

6.90 

6.38 

4.29 

Grades 1-11 

1,110.2 

2.98 

2.98 

2.81 

2.04 

9.59 

8.69 

6.75 

12th grade (no diploma) 

1,110.2 

5.68 

5.68 

5.60 

3.34 

12.09 

11.21 

5.79 

High school (or equivalent) 

1,732.5 

1.60 

1.60 

1.45 

1.15 

3.33 

2.90 

2.90 

Regular high school diploma 

1,477.5 

1.88 

1.88 

1.59 

1.30 

3.47 

2.93 

3.09 

GED or alternative credential 

852.1 

5.17 

5.17 

4.90 

3.71 

12.68 

10.58 

10.89 

Some college or associate's degree 

1,796.0 

2.00 

2.00 

1.88 

1.70 

2.97 

2.63 

2.75 

Some college credit, but less than 
1 year 

862.9 

3.91 

3.91 

3.60 

3.61 

6.39 

6.29 

5.24 

1 or more years of college credit, 
no degree 

1,572.7 

3.07 

3.07 

2.76 

2.46 

4.54 

3.43 

3.72 

Associate's degree 

700.2 

3.02 

3.02 

3.09 

3.34 

5.33 

4.07 

4.89 

Bachelor's degree 

968.7 

2.17 

2.17 

2.10 

1.03 

2.21 

1.53 

1.71 

Graduate or professional degree 

953.0 

2.42 

2.42 

2.54 

1.49 

2.42 

1.36 

2.06 

Master's degree 

875.0 

3.06 

3.06 

3.10 

1.82 

3.08 

1.81 

2.46 

Professional degree beyond a 
bachelor's degree 

372.7 

4.62 

4.62 

4.75 

4.97 

6.13 

1.59 

6.00 

Doctorate degree 

449.6 

6.01 

6.01 

5.95 

3.03 

5.58 

0.77 

5.61 


See notes at end of table. 
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Table D-1 . Standard errors for Table 2-3: Number and percentage distribution of respondents who reported any 

credentials, percentage of adults who reported any certification/license or certificate, and among those 
who reported any credentials, percentage distribution who reported a certification/license only, a 
certificate only, or both credentials, by respondents characteristics: 2010-11 — Continued 

Among those who reported any credentials, 
Percentage distribution of Percentage of respondents percentage distribution of respondents who 

respondents who reported who reported reported 

Both a 

Any certification/ 


Respondent characteristics 

Number (in 
thousands) 

No 

credentials 

Any 

credentials 

certification/ 

license 

Any 

certificate 

Certification/ 
license only 

Certificate 

only 

license and 
certificate 

Labor force status 

Employed 

1,687.8 

1.24 

1.24 

1.17 

0.94 

1.88 

1.44 

1.47 

Unemployed (seeking 
employment) 

1,460.2 

3.28 

3.28 

2.67 

2.55 

7.12 

5.93 

5.67 

Not in labor force 

1,574.8 

1.36 

1.36 

1.19 

0.95 

2.77 

2.69 

1.74 

Labor force status unknown 

532.8 

8.64 

8.64 

8.75 

7.32 

13.35 

4.66 

13.23 

Immigration status 

Born in U.S. 

1,452.0 

0.95 

0.95 

0.89 

0.69 

1.63 

1.26 

1.30 

Born outside of U.S. 

1,452.0 

3.51 

3.51 

3.14 

2.19 

4.99 

3.53 

3.94 


# Rounds to zero. 

1 Black includes African American, Other includes Pacific Islanders and American Indians (including Alaska Natives), and two or more races includes non-Hispanics reporting 
more than one race. Race categories exclude persons of Hispanic ethnicity. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education Survey (ATES) Pilot Study, 2010-11. 
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2010 Adult Training and Education Survey Pilot Study 

ATES Questionnaire 


Contents [Ctrl +Click to follow link] 

Introduction E-4 

Certification/license and Licensure E-6 

Education Attainment Items E-1 1 

Certificates E-1 2 

Additional Background and Labor Force Participation E-1 5 


SA_NAME = Sampled Adult’s name used for Proxy Approach 2 (1 adult sampled) 

PR_NAME = Proxy subject’s name used for Proxy Approach 1 (2 adults sampled) 

TEXT 1 = your / SAJSIAME’s / PR_NAME’s 

TEXT 2 = you / SAJMAME / PR_NAME 

TEXT 3 = Were you / Was (SA_NAME/PR_NAME) 

TEXT4 = Are you / Is (SANAME/PRJMAME) 

TEXT 5 = Do you / Does (SA_NAME/PR_NAME) 

TEXT6 = you / he / she 

TEXT7 = you have / (SANAME/PRNAME) has 
TEXT 8 = Have you / Has (SA_NAME/PR_NAME) 

TEXT9 = your / his / her 

TEXT10 = yourself / himself / herself 

TEXT 11= Were you / Was (he / she) 

TEXT12 = Do you / Does (he / she) 

TEXT13 = Were you/ Was SA J PR Name 
TEXT14 = do you/ does SA/ PR Name 
TEXT15 = are you/ is SA/ PR Name 
TEXT16 = have you/ has SA/ PR Name 
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Introduction 


INI . Hello. My name is from Westat. I’m calling about a research study 

sponsored by the U.S. Department of Education. Your household recently filled out a short 
survey for us and I am calling to ask some additional questions about [SA_NAME’s] training 
and education. (May I please speak to [SA_NAME] [the SEX in your household who is age 
AGE]?) 


YES, R SPEAKING 1 (GOTO IN3) 

YES, R AVAILABLE 2 (GOTO IN2) 

NO, R NOT AVAILABLE* 3 (GOTO IN2A) 

NO, R DOES NOT LIVE HERE 4 (GOTO RESULT) 

NO, DOES NOT KNOW R 5 (GOTO RESULT) 


* For some of these cases, proxy interviews were conducted and “your” was changed 
to [NAME] or [HIS/HER]. 

IN2. Hi [SA_NAME], My name is from Westat. I’m calling about a research 

study sponsored by the U.S. Department of Education. Your household recently filled out a 
short survey for us and I am calling to ask some additional questions about your training and 
education. (GO to in3) 

IN2A. IF PROXY APPROACH 2 AND CALLING ALGORITHM COMPLETE AND THIS IS FINAL 
CALL GO TO IN2C, OTHERWISE READ: 

When would be a good time to call back? SET APPOINTMENT 

IN2B. Is this the best phone number to reach [SA_NAME] [you]? 


YES 1 (GOTO RESULT) 

NO 2 (RECORD NEW NUMBER) 

REFUSED -7 (GOTO RESULT) 

DON’T KNOW -8 (GO TO RESULT) 


IN2C. To whom am I speaking? 

CAT I DISPLAYS NAME, SEX, AGE FOR EACH ELIGIBLE PROXY RESPONDENT (EP_1 , EP_2, 
EP_3) 


EP_1 

EP_2 

EP_3 

SOMEONE ELSE 

REFUSED 

DON’T KNOW .... 


. 1 (GOTO IN2D) 

.2 (GOTO IN2D) 

.3 (GOTO IN2D) 

.4 (GOTO RESULT) 
-7 (GO TO RESULT) 
-8 (GO TO RESULT) 
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IN2D. We have been trying to reach SA_NAME. Since [he/she] is not available I would like to ask 
you some questions about [SA_NAME]’s education and training. On average, the interview 
takes about 15 minutes depending on your responses, and your participation is voluntary. 
Your answers may be used only for statistical purposes and may not be disclosed, or used, 
in identifiable form for any other purpose except as required by law. [You will receive 
[$10/$20] for completing the interview.] Is this a good time for you to continue? 

CONTINUE 1 (GO TO IN4) 

NOT A GOOD TIME 2 (GO TO RESULT) 

RESPONDENT ANSWERS FIRMLY CANNOT ANSWER PROXY 

QUESTIONS 3 (GO TO RESULT) 

IN3. On average, the interview takes about 15 minutes depending on your responses, and your 
participation is voluntary. Your answers may be used only for statistical purposes and may 
not be disclosed, or used, in identifiable form for any other purpose except as required by 
law. [You will receive [$10/$20] for completing the interview.] Is this a good time for you to 
continue? 


YES 1 (CONTINUE) 

NO 2 (SET CALLBACK) 


IN4. Let’s get started. As we go along, if you don’t know an answer to any question, you can just 
say “don’t know.” First, I have a few questions about [TEXT1] work experience. 

IN5. Did [TEXT2] work at a job for pay or income at any time in the past 12 months, including self- 
employment? 


yes 

no 

REFUSED 

DON’T KNOW 


. 1 (GOTO IN6) 
.2 (GOTO CNl ) 
-7 

-8 (GO TO CNl) 


IN6. [TEXT13] employed by a private company, government, a nonprofit organization, or 

[TEXT 1 1 ] self-employed? [code all that apply - people employed by a person or in a 

PRIVATE HOME = PRIVATE COMPANY] 


PRIVATE COMPANY 1 (GOTO IN7) 

GOVERNMENT 2 (GOTO IN6A) 

NONPROFIT ORGANIZATION 3 (GOTO IN7) 

SELF EMPLOYED 4 (GOTO IN7) 

REFUSED -7 (GOTO IN7) 

DON’T KNOW -8 (GO TO IN7) 

IN6A. Would that be the federal, state, or local government? 

FEDERAL 1 

STATE 2 

LOCAL (COUNTY, CITY, TOWNSHIP) 3 

REFUSED -7 

DON’T KNOW -8 
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IN7. [TEXT 4] currently employed? 


yes 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 


Certification and Licensure 


CN1 . Now I’d like to ask you about professional certification and licensure. 

[TEXT5] have a professional certification or a state or industry license? 

Probe: A professional certification or license shows you are qualified to perform a specific job 
and includes things like Licensed Realtor, Certified Medical Assistant, Certified Construction 
Manager, a Project Management Professional or PMP certification, or an IT certification. 


yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


(GOTOCNlA) 
(GOTO EDI) 
(GOTO EDI) 
(GOTO EDI) 


CNPRO. [DID YOU USE THE PROBE ON CN1?] all cnl come here before doing above skip 


YES 1 

NO 2 


CN1A. [TEXT12] have more than one certification or license? 


yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


(GOTO CNlB) 
(GOTO CN3) 
(GOTO CN3) 
(GOTO CN3) 


CN1B. How many [TEXT14] have? TWO DIGITS 


CN2. Let’s start with [TEXT1] most recent certification or license. 


loop: ask cn3 to cn6 for each certification/license, take up to five. 


if cn1a=2, fill cn3 with “[TEXT1]” instead of “this” 

CN3A. What is the name of [TEXT1] certification or license? 

VERBATIM TEXT STRING [USE LENGTH OF NHES 2005 OR 2007 ADULT ED STRING] 


REFUSED -7 

DON’T KNOW -8 


CN4. Is that a certification, a license, or both? if clear from name, do notask 


CERTIFICATION 1 

LICENSE 2 

BOTH 3 

REFUSED -7 

DON’T KNOW -8 
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CN3. What kind of work is this [certification/license] for? 


VERBATIM TEXT STRING 


REFUSED -7 

DON’T KNOW -8 


IF CN4=3 THEN FILL WITH CERTIFICATION 

CN5. In what year did [TEXT6] get it? 

[ENTER FOUR DIGIT YEAR] RANGE: 1925-20 10 


REFUSED -7 

DON’T KNOW -8 


IF CN4=3 THEN FILL WITH CERTIFICATION 

CN6. Did [TEXT2] get this [certification] [license] mainly for work related reasons or mainly for 
personal interest? 


MAINLY WORK-RELATED 1 

MAINLY PERSONAL INTEREST 2 

REFUSED -7 

DON’T KNOW -8 


IF CN4=3 THEN FILL WITH CERTIFICATION 


IF CNlA> or =1 : GOTO cn6a to ask about most recent certification/license at cn5 (if 

MORE THAN ONE IN A YEAR, TAKE FIRST LISTED AT CN3A) THAT IS WORK-RELATED 
(CN6 = 1 ). IF NONE ARE WORK-RELATED, ASK ABOUT MOST RECENT PERSONAL 
INTEREST OR REFUSED OR DON’T KNOW (CN6=2, -7,-8). 

CN7. These next questions are about the training and education [TEXT2] took in order to earn 
the [CERTIFICATION/LICENSE NAME][certification] [license]. 

Did [TEXT6] take courses or training to earn the [CERTIFICATION/LICENSE NAME] 
[certification] [license]? 


1 (goto cn7a) 

2 (GOTO CN8n) 

7 (GOTO CN7a) 

8 (GOTO CN8N) 

if cn4=3 then fill with certification 


YES 

NO 

REFUSED 

DON’T KNOW 
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CN7A. Did [TEXT6] take the coursework or training for the [certification] [license] as part of a 
college or university certificate or degree program? 


yes 

no 

REFUSED 

DON’T KNOW 


. 1 (GOTOCN7B) 
.2 (GOTO CN7C) 
-7 (GOTO CN7C) 
-8 (GOTO CN7C) 


IF CN4=3 THEN FILL WITH CERTIFICATION 


CN7B. Was that a certificate program, an associate’s degree program, a bachelor’s degree 
program, or something else? 


associate’s 1 

BACHELOR’S 2 

CERTIFICATE 3 

OTHER 91 

SPECIFY 

REFUSED -7 

DON’T KNOW -8 


CN7C. Was the coursework or training mainly self-study or mainly classes or courses with an 
instructor? 


MAINLY SELF-STUDY 1 

MAINLY INSTRUCTOR 2 

REFUSED -7 

DON’T KNOW -8 


IFCN7B=1 OR 2, GOTOCN10 

CN8. How many weeks or months did it take [TEXT2] to earn the [certification] [license], 
including coursework and other training? 

TWO DIGITS — ENTER ‘00’ IF LESS THAN 1 WEEK. RANGE: 00-99 
IF CN8= 00,-7 OR -8, SKIP UNIT 
UNIT 1 -WEEK 
2=MONTH 


REFUSED -7 

DON’T KNOW -8 


IF CN4=3 THEN FILL WITH CERTIFICATION 
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CN9. About how many hours per week or per month did [TEXT2] spend with an instructor? 


TWO DIGITS 

NOTE TO interviewer: online instructor is acceptable, range: 0-99 
0 , -7, -8 SKIPS UNIT 


REFUSED -7 

DON’T KNOW -8 


[ONLINE INSTRUCTOR IS ACCEPTABLE] 

IF CN7 = 1 OR -7 GO TO CN10. ONLY ASK CN9A IF CN7 = 2 OR -8 

CN9A. About how many hours per week or per month did it take [text2] to earn the certification? 


two digits range: 1-99 

IF CN8=- 7 OR -8, SKIP UNIT 
UNIT 1 —WEEK 
2=MONTFI 

REFUSED -7 

DON’T KNOW -8 

CN10A. Did [TEXT6] have to do any of the following to get this [certification] [license]... 

a. demonstrate skills while on the job? yes no 

b. pass a test or exam? yes no 

c. submit a portfolio of work? yes no 

REFUSED -7 

DON’T KNOW -8 


IF CN4=3 THEN FILL WITH CERTIFICATION 
CN1 1 . [TEXT5] have to take... 


a. continuing education classes or earn CEUs to maintain it? yes no 

b. to take periodic tests? yes no 


REFUSED -7 

DON’T KNOW -8 


IF in7=1 (currently employed) goto cn12, else goto ON 14 


CN12. Is this [certification] [license] related to [TEXT1] current job? 


yes 

NO 

REFUSED 

DON’T KNOW 


. 1 (GOTO CNi 3) 
2 (GOTO CNi 4) 
-7 (GOTO CNi 4) 
-8 (GOTO CNi 4) 
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CN13. Is it required for [TEXT9] current job? 


yes 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 


CN14. [TEXT3] [certified] [licensed] by [TEXT9] state, industry, a company, a professional 
association, or some other organization? mark all that apply 


state 1 

INDUSTRY 2 

COMPANY 3 

PROFESSIONAL ASSOCIATION 4 

OTHER 91 

SPECIFY 

REFUSED -7 

DON’T KNOW -8 


CN15a. Can this [certification] [license] be... 

a. revoked or suspended for any reason? yes no 

b. used if [TEXT2] wanted to get a job with any employer 
in that field? 

[CERTIFICATIONS/LICENSES AND LICENSES THAT ARE RECOGNIZED 
STATE-WIDE SHOULD BE RECORDED AS “YES.”] YES NO 

REFUSED -7 

DON’T KNOW -8 


IF CN4=3 THEN FILL WITH CERTIFICATION 


CN16. What benefits did [TEXT2] receive from earning the [certification] [license]? mark all that 

APPLY 


PROMOTION 1 

HIGHER PAY OR BONUS 2 

CAREER ADVANCEMENT 3 

IS JUST MANDATORY OR REQUIRED FOR JOB 4 

IMPROVED JOB PERFORMANCE 5 

HELPED ME STAY CURRENT WITH NEW REGULATIONS, 

LAWS, OR TECHNOLOGIES 6 

TO CHANGE JOB OR CAREER FIELD, ENTER THE WORKFORCE, 

OR START OWN BUSINESS 7 

NO BENEFITS RECEIVED 8 

OTHER 91 

SPECIFY 

REFUSED -7 

DON’T KNOW -8 


IFCNl3=1 GOTOEDINT, ELSE ASK CNi 7 
IF cn4=3 then fill with certification 
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CN17. Is the [certification][license] currently valid or has it expired? 


CURRENTLY VALID 1 (GOTEDl) 

EXPIRED 2 (GOTO CNl 8) 

REFUSED -7 (GOTO EDI ) 

DON’T KNOW -8 (GOTO EDINT) 

IF CN4=3 THEN FILL WITH CERTIFICATION 

CNl 8. [TEXT12] plan to renew it? 

YES 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 

Education Attainment Items 

EDI . These next questions are about other education or schooling [TEXT2] may have. 

[TEXT4] currently attending or enrolled in school? 

yes 1 (goto ed2) 

NO 2 (GOTOED3) 

REFUSED -7 (GOTO ED3) 

DON’T KNOW -8 (GOTOED3) 

ED2. What grade or year of school [TEXT 1 5] attending? 

ELEMENTARY GRADES 1-8 1 

HIGH SCHOOL GRADES 9-12 2 

VOCATIONAL, TECHNICAL, CERTIFICATE, OR 

BUSINESS SCHOOL BEYOND HIGH SCHOOL LEVEL 3 

UNDERGRADUATE COLLEGE OR UNIVERSITY 4 

GRADUATE OR PROFESSIONAL SCHOOL 5 

REFUSED -7 

DON’T KNOW -8 

ED3. What is the highest degree or level of school [TEXT7] completed? 

NO SCHOOLING 0 

NURSERY SCHOOL N 

KINDERGARTEN K 

GRADES 1-1 1 1 

1 2TH GRADE NO DIPLOMA 2 

REGULAR HIGH SCHOOL DIPLOMA 3 

GED OR ALTERNATIVE CREDENTIAL 4 

SOME COLLEGE CREDIT, BUT LESS THAN 1 YEAR OF 

COLLEGE CREDIT 5 

1 OR MORE YEARS OF COLLEGE CREDIT, NO DEGREE 6 

ASSOCIATE’S DEGREE (FOR EXAMPLE: AA, AS) 7 

BACHELOR’S DEGREE (FOR EXAMPLE: BA, BS) 8 
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MASTER’S DEGREE (FOR EXAMPLE: MA, MS, MENG, MED, 


MSW, MBA) 9 

PROFESSIONAL DEGREE BEYOND A BACHELOR’S 

DEGREE (FOR EXAMPLE: MD, DVS, DDM, LLB, JD) 10 

DOCTORATE DEGREE (FOR EXAMPLE: PHD, EDD) 11 

REFUSED -7 

DON’T KNOW -8 


ED4. Interviewer only: Did the respondent say they had a “certificate” or some type of 

TRADE OR VOCATIONAL DIPLOMA? 


YES, CERTIFICATE 1 (GOTO ED5C) 

YES, TRADE OR VOCATIONAL DIPLOMA 2 (GOTO ED5A) 

NO 3 (GOTOCTl) 


ED5A. Is that diploma the equivalent of a high school diploma or something else? 


HS DIPLOMA EQUIVALENT 

SOMETHING ELSE 

REFUSED 

DON’T KNOW 


.1 (GOTOCTINT) 
.2 (GOTOED5B) 
-7 (GOTOCTl) 
-8 (GOTOCTl) 


ED5B. We will refer to these types of vocational or trade diplomas as certificates in the next set of 
questions. 


ED5C. These next questions are about certificates. 

Certificates 


CT1 . CTINT. Some people decide to enroll at a college, university, community college, or trade 
school to earn a certificate rather than a degree. 


[TEXT8] ever earned this type of certificate? 


Probe: An educational certificate is typically earned by completing a program of study offered 
by a college or university, a community college, or a trade school, but it does not lead to an 
associate’s, bachelor’s, or graduate degree. Sometimes these are also called vocational 
diplomas, for example, a cosmetology or mechanics diploma, which differs from a high 
school diploma. 

yes 1 (goto ct2) 

NO 2 (GOTO ALl) 

REFUSED -7 (GOTO ALl) 

DON’T KNOW -8 (GOTO ALl ) 

CTPRO. DID YOU USE PROBE ON CT1 all ctl’s come here before skipping 


YES 1 

NO 2 

CT2. [TEXT12] have more than one certificate? 

YES 1 (GOTO CT2a) 

NO 2 (GOTO SKIP AT CT2C) 

REFUSED -7 (GOTO SKIP AT CT2C) 

DON’T KNOW -8 (GOTO SKIP AT CT2C) 
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CT2A. HOW MANY [TEXT 1 4] HAVE? □□ (GOTO CT2B) 

CT2B. Let’s start with the most recent certificate. 

loop: ask ct2c to ct6 for each certificate, take up to five certificates. 

IF CNl =1 THEN ASK CT2C FOR EACH CERTIFICATE. 

CT2C. Is (this/your) certificate part of the coursework or training [TEXT2] took for a certification or 
license? 

[DISPLAY ONLY ON LOOPS 2-5: The next set of questions refers to [TEXT9] next most recent 


certificate.] 

yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


CT3. What is the major subject or field of study for this/text 9 certificate? 
VERBATIM TEXT STRING 


REFUSED -7 

DON’T KNOW -8 


CT4. What is the name of this certificate? 

VERBATIM TEXT STRING [USE LENGTH OF NHES 2005 OR 2007 ADULT ED QUEX STRING] 


REFUSED -7 

DON’T KNOW -8 

CT5. Did [TEXT6] complete this certificate program? 

yes 1 

NO 2 (GOTO CT6 BOX) 

REFUSED -7 (GOTO CT6 BOX) 

DON’T KNOW -8 (GOTO CT6 BOX) 


CT6. In what year did [TEXT6] complete it? 

[ENTER FOUR DIGIT YEAR] RANGE: 1925-2010 


REFUSED -7 

DON’T KNOW -8 


IF CT2 = 1 : GOTO CT8 TO ASK ABOUT MOST RECENT CERTIFICATE (CT6) THAT IS COMPLETED 
(CT5=1 ). 
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CT8. These next questions are about the [CERTIFICATE NAME] certificate. 
What type of school or organization provided the certificate program? 
PROBE FOR CATEGORY 


A COMMUNITY COLLEGE 1 

ANOTHER UNIVERSITY OR COLLEGE 2 

A TRADE SCHOOL 3 

BUSINESS OR COMPANY 4 

PROFESSIONAL ASSOCIATION 5 

TRADE UNION 6 

NONPROFIT ORGANIZATION 7 

A FEDERAL, STATE, OR LOCAL GOVERNMENT 8 

SOMEPLACE ELSE! 91 

SPECIFY 

REFUSED -7 

DON’T KNOW -8 

CT9. How many credits or hours were required to complete the certificate program? 

TWO DIGITS □□ (IF ZERO GOTO CTlO) 

REFUSED -7 (GOTOCTlO) 

DON’T KNOW -8 (GOTOCTlO) 

CT9A. Is that credits or actual hours? 

CREDITS/CREDIT HOURS 1 

HOURS 2 


CT1 0. Did [TEXT2] take the education or training to earn continuing education units or CEUs? 


yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


IF CT2C=1 OR -9 GOTO ALl . 


CT1 1 . Was the training mainly self-study or mainly classes or courses with an instructor? 


MAINLY SELF-STUDY 1 

MAINLY INSTRUCTOR 2 

REFUSED -7 

DON’T KNOW -8 


[IFIN5 = 2 OR IN7=2 GOTO CT14] 

CT12. Is this certificate related to [TEXT1] current job? 

1 

2 (GOTO CT14) 

7 (GOTO CT14) 

8 (GOTO CT14) 


YES 

NO 

REFUSED 

DON’T KNOW 
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CT1 3. Is it required for [TEXT9] current job? 


yes 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 

CT14. What benefits did [TEXT2] receive from earning the certificate? mark all that apply 

promotion 1 

HIGHER PAY OR BONUS 2 

CAREER ADVANCEMENT 3 

IS JUST MANDATORY OR REQUIRED FOR JOB 4 

IMPROVED JOB PERFORMANCE 5 

HELPED ME STAY CURRENT WITH NEW REGULATIONS, 

LAWS, OR TECHNOLOGIES 6 

NO BENEFITS RECEIVED 8 

OTHER 91 

SPECIFY 

REFUSED -7 

DON’T KNOW -8 


Additional Background and Labor Force Participation (From CPS and NHES) 

AL1. Now I would like to ask you a few more questions about [TEXT1] background. In what 
year[TEXT13] born? 


YEAR 19DD range:00-93 

REFUSED -7 

DON’T KNOW -8 

AL3. [TEXT4] of Spanish, Hispanic, or Latino origin? 

yes 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 


AL4. Which of the following races [TEXT14] consider [TEXT10] to be? You may name more 
than one. 

[if “Hispanic” probe: Is that White Hispanic, Black Hispanic, both, or something else?] 

[CODE ALL THAT APPLY] 

[TEXT4]... 


White, 1 

Black or African American, 2 

American Indian or Alaska Native, 3 

Asian, or 4 

Native Hawaiian or other Pacific Islander? 5 

REFUSED -7 

DON’T KNOW -8 


Wording experiment - Marital status (Ask AL5V1 for even phone numbers, AL5V2 and AL5V2A for 
odd phone numbers) 
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LEVOD (1=EVEN, 2=ODD) 
AL5V1 . [TEXT4] currently... 


Married 1 

In a registered domestic partnership or 

CIVIL UNION, 2 

Widowed, 3 

Divorced, 4 

Separated, or 5 

Never married? 6 

REFUSED -7 

DON’T KNOW -8 

AL5V2. [TEXT4] currently... 

Married 1 

Widowed, 3 

Divorced, 4 

Separated, or 5 

Never married? 6 

REFUSED -7 

DON’T KNOW -8 

AL5VA.[TEXT4] currently living in a registered domestic partnership or 

yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


IF AL5VA= 1 skip AL50V 

AL50V. [TEXT4] currently living with a partner? 


yes 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 

AL6. [RECORD SA_NAME SEX; CONFIRM IF NECESSARY] 

MALE 1 

FEMALE 2 

REFUSED -7 

DON’T KNOW -8 


(GO TO AL5VA) 

(GO TO AL5VA) 
(GO TO AL50V) 
(GO TO AL50V) 
(GO TO AL50V) 
(GO TO AL50V) 
(GO TO AL50V) 
(GO TO AL50V) 


civil union? 

(GO TO AL6) 
(GO TO AL50V) 
(GO TO AL50V) 
(GO TO AL50V) 
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AL8. In what state, country, or territory [TEXT13] born? 

ONE OF 50 STATES OR THE DISTRICT OF COLUMBIA 1 (GO TO BOX BEFORE ALl 1 ) 

ONE OF THE U.S. TERRITORIES [PUERTO RICO, GUAM, 

AMERICAN SAMOA, U.S. VIRGIN ISLANDS, MARIANA 

ISLANDS, MIDWAY ISLANDS, OR SOLOMON ISLANDS] 91 (GO TO AL80V) 

SPECIFY 

SOME OTHER COUNTRY 92 (GO TO AL80V) 

SPECIFY 


REFUSED -7 (GOTO BOX AT ALl 1 ) 

DON’T KNOW -8 (GOTO BOX AT ALl 1 ) 


AL80V. How old [TEXT13] when [TEXT6] first moved to the (United States/50 states or the 
District of Columbia)? 


AGE □□ range:00-99 

REFUSED -7 

DON’T KNOW -8 


If AL8QV > 6, go to AL9. Else, go to box before ALl 1. 


AL9. What is the highest degree or level of school [TEXT2] completed before moving to the U.S.? 


NO FORMAL EDUCATION COMPLETED 0 

SOME PRIMARY SCHOOL 1 

PRIMARY SCHOOL COMPLETED 2 

SECONDARY SCHOOL COMPLETED 3 

SOME UNIVERSITY-LEVEL EDUCATION, WITHOUT DEGREE 4 

UNIVERSITY-LEVEL EDUCATION, WITH DEGREE 5 

REFUSED -7 

DON’T KNOW -8 


If IN5 = 2 (did not work in the past 12 months), 
autocode ALII -2 and ALl 2 = 2 and go to box before 
ALl 4. Else, go to ALII. 


ALl 1 . During the past week , did [TEXT2] work at a job for pay or income, including 
self-employment? 


yes 1 (go to aLI 3) 

NO 2 (GOTOAL12) 

RETIRED 3 (GO TO AL14) 

DISABLED/UNABLE TO WORK 4 (GO TO ALl 4) 

REFUSED -7 (GOTO ALl 3) 

DON’T KNOW -8 (GOTO ALl 3) 


ALl 2. [TEXT3] on leave or vacation from a job during the past week ? 


yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


(GO TO AL13) 
(GO TO AL14) 
(GOTO ALl 3) 
(GOTO ALl 3) 
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AL1 3. About how many total hours per week [TEXT 1 4] usually work for pay or income? 
[IF HOURS VARY, PROBE FOR AVERAGE PER WEEK.] 

I I I WEEKLY HOURS RANGE: 1 -90 


REFUSED -7 

DON’T KNOW -8 


If IN5 = 2 (did not work in the past 12 months), then 
autocode AL14 = 0 and go to AL15. Else, go to AL14. 


AL14. In the past 12 months, how many months [TEXT16] worked for pay or income? 

| | | range:0-12 

MONTHS 


REFUSED -7 

DON’T KNOW -8 


IfAL1 1 = 1 (worked last week) or AL12 = 1 (on leave or 
vacation), go to AL21. Else if ALII = 3 (retired), then 
autocode AL17 = 3 and go to AL21. Else if ALII = 4 
(unable to work), then autocode AL17 = 5 and go to 
AL21A. Else, gotoAL15. 


AL1 5. [TEXT8] been actively looking for work in the past 4 weeks ? 

1 (goto alI 6) 

2 (goto alI 7) 

7 (goto alI 7) 

8 (goto alI 7) 


YES 

NO 

REFUSED 

DON’T KNOW 


ALI 6. What [TEXT16] been doing in the past 4 weeks to find work? [TEXT8]... 


YES NO 

a. Checked with an employment agency? 1 2 

b. Checked with an employer directly or sent a resume? 1 2 

c. Checked with friends or relatives? 1 2 

d. Placed or answered job ads? 1 2 


If any of ALI 6 a-d = 1 (actively looking for work), go to 
box after ALI 7. Else, go to ALI 7. 










AL17. What [TEXT1 3] doing most of last week? Would you say... 


Keeping house or caring for children or other 


DEPENDENTS, 1 

Going to school 2 

Retired, 3 

Volunteering, 4 

U NABLE TO WORK, OR 5 

Something else? 91 

What was that? 

REFUSED -7 

DON’T KNOW -8 


If IN5 = 2 (not worked in past 12 months), then go to 
AL18. Else, gotoAL21. 


AL1 8. [TEXT8] ever worked at a job for pay or income? 


yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


(GO TO AL19) 
(GO TO AL20) 
(GOTO ALl 9) 
(GOTO ALl 9) 


ALl 9. In what year did [TEXT6] last work for pay or income? 

| | | | | range: 1925-2010 

YEAR 


REFUSED -7 

DON’T KNOW -8 


AL20. [TEXT5] plan to work at a job for pay or income in the next year? 


yes 1 

no 2 

REFUSED -7 

DON’T KNOW -8 


(GO TO AM12) 
(GO TO AM12) 
(GOTO AM12) 
(GOTO AM12) 


AL21 A. For whom did [TEXT2] work at [TEXT9] current or most recent job? 

[IF MORE THAN ONE CURRENT JOB, ASK FOR THE COMPANY THEY HAVE 
WORKED FOR THE LONGEST.] 


NAME OF COMPANY 


REFUSED -7 

DON’T KNOW -8 
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AL21 B. What kind of business or industry is this? 


[business/industry probe: For example, TV and radio manufacturing, retail 
shoe store, state labor department, or farm.] [if more than one job, collect job 

WHERE R HAS WORKED THE LONGEST.] 

TYPE OF INDUSTRY 


REFUSED -7 

DON’T KNOW -8 


If IN6 = 4 (self-employed in the past 12 months), go to 
AL22. Else, go to AL23. 


AL22. [IS THIS [TEXT9] OWN BUSINESS?] 


YES 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 


AL23. What kind of work [(are/were) you/(is/was) SA_NAME] doing and what (are/were) 
[TEXT 1 ] most important activities or duties? 

[job probe: For example, electrical engineer, stock clerk, typist, or farmer.] 
[important duty probe: For example, typing, keeping account books, filing, 
selling cars, operating printing press, or finishing concrete.] 

[IF MORE THAN ONE JOB, COLLECT JOB WHERE R HAS WORKED THE LONGEST.] 

KIND OF WORK 

IMPORTANT DUTY 

IMPORTANT DUTY 


REFUSED -7 

DON’T KNOW -8 


AL25. (If [TEXT2] had worked for all 12 months this past year,) About how much (would 
(TEXT6/TEXT14) have earned/(TEXT5)earn) before taxes and other deductions at 
(NAME OF COMPANY/TEXT9 business)? 


amount $nnn,nnnRANGE:6. 00 - 200, 000 

Per 

HOUR 1 

DAY 2 

WEEK 3 

Bl WEEKLY 4 

MONTH 5 

YEAR 6 

OTHER 91 

What (is/was) that? 

REFUSED -7 

DON’T KNOW -8 
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AL26. Does [TEXT9] occupation have legal or professional requirements for continuing training 
or education? 


yes 1 

no 2 


IF AL26=1 AND CNl -2 GOTO AL26A, ELSE GOTO AMl 2 


AL26A. [TEXT5] have a qualification that meets those requirements? 


Yes 1 

No 2 

REFUSED -7 

DON’T KNOW -8 


(GOTO AL26B) 
(GOTO AMl 2) 
(GOTO AMl 2) 
(GOTO AMl 2) 


AL26B. What is the name of that qualification? 


NAME 

REFUSED -7 

DON’T KNOW -8 


[IF PROXY APPROACH 1 AND ASKING ABOUT PROXY SUBJECT GO TO 
AM4] 

AM 12. What was the total income of all persons in [TEXT1] household over the past year, including 
salaries or other earnings, interest, retirement, and so on for all household members? 

Was it... 

IF AL25>$25,000 GOTO AMl 20V 

$25,000 OR LESS, OR 1 (READ SET 1) 

More than $25,000? 2 (gotoam12ov) 

If AL25>$50, 000 GOTO SET3 

AM 120V. Was it... 

$50,000 OR LESS, OR 

More than $50,000? .... 

Was it.... 

[SET 1] 

$5,000 OR LESS 

$5,001 TO $10,000 

$10,001 TO $15,000 

$15,001 TO $20,000, OR 
$20,001 TO $25,000? .... 

[SET 2] 

$25,001 TO $30,000 

$30,001 TO $35,000 

$35,001 TO $40,000 

$40,001 TO $45,000, OR 
$45,001 TO $50,000? .... 


[SET 3] 

$50,001 TO $60,000, 11 

$60,001 TO $75,000, 12 

$75,001 TO $100,000, OR 13 

Over $100, 000? 14 


1 (READ SET 2) 

2 (READ SET 3) 


1 

2 

3 

4 

5 


..6 

..7 

..8 

..9 

10 
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REFUSED -7 (GOTO AM3) 

DON’T KNOW -8 (GOTO AM3) 

AM3. [TEXT5]... 

Own (his/her) home, 1 

Rent (his/her) home, or 2 

Have some other arrangement? 3 

REFUSED -7 

DON’T KNOW -8 


AM3A. Now, I’d like to confirm your address I have. . . 
STREET ADDRESS 


[RECORD CHANGES TO STREET OR PRESS ENTER IF NO CHANGES] 

CITY 

[RECORD CHANGES TO CITY OR PRESS ENTER IF NO CHANGES] 

STATE 

[RECORD CHANGES TO STATE OR PRESS ENTER IF NO CHANGES] 

ZIP CODE 

[RECORD CHANGES TO ZIP OR PRESS ENTER IF NO CHANGES] 

IF PROXY APPROACH 1 AND ASKING ABOUT SAMPLED ADULT ASK CT15, OTHERWISE 
GO TO AM4. 

CT15. Now I would like to ask you about education or training [PR_NAME] may have had. 


CONTINUE 1 (GO TO IN5) 

RESPONDENT FIRMLY INDICATES CANNOT ANSWER 

PROXY QUESTIONS 2 

[IF R INDICATES THEY HAVE NEVER HEARD OF [PR_NAME] OR 
[PR_NAME] DOES NOT LIVE IN THE HOUSEHOLD THEN SELECT 
“RESPONDENT FIRMLY INDICATES CANNOT ANSWER PROXY 
QUESTIONS” AND FILL OUT A PROBLEM SHEET] 
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AM4. [IF NO INCENTIVE GO TO CLOSE] Those are all the questions I have. Before I hang up 
I need information to prepare your ($10/$20) check. How should your name appear on 
the check? 

FIRST NAME: Ml: LAST NAME: 

AM4A. Should I use the address: 

[DISPLAY CONFIRMED ADDRESS]? 

CLOSE. Those are all the questions I have for you./You should receive your check within the next 2 
weeks. Thank you for your time. 
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Introduction and Methodology 


The Education Statistics Services Institute of the American Institutes for Research (AIR) is 
helping the National Center for Education Statistics (NCES) develop a government strategy for 
measuring the prevalence of subbaccalaureate credentials in the United States. As a first step, 
AIR partnered with Shugoll Research to conduct focus groups with individuals who hold less 
than a bachelor’s degree and work in fields that may offer subbaccalaureate credentials. The 
purpose of this phase is to understand the language used to describe these credentials and their 
common attributes. 

Shugoll Research conducted a total of three focus groups in suburban Washington, DC, on 
December 17 and 18, 2009. The groups were held in a specially designed focus group facility 
with a one-way mirror that allowed for observation of the groups and the opportunity to provide 
input to the moderator. Two focus groups were with individuals who self-identified as being 
certified, registered, or licensed in the fields of infonnation technology, health care, or business. 
These fields were selected because they represent areas where significant certification exists. 

One focus group was held with individuals who self-identified as holding certifications in other 
fields or having interest in becoming certified in the future. Respondents in all groups were 
required to be 21-40 years old with at least a high school degree or GED but less than a 
bachelor’s degree. Participants could have completed some college or obtained an associate’s 
degree. They could not have participated in a focus group within the last 6 months and must have 
described themselves as being articulate and comfortable expressing themselves in a group. 

Respondents were recruited by telephone between December 3 and 16, using a database of 
potential participants maintained by Shugoll Research. Each prospective respondent was 
screened using a recruitment screener that is shown in attachment F-l. If they were qualified, 
respondents were offered a $75 honorarium to encourage participation. The screener shows that, 
originally, people in the construction industry were to be included in the information technology/ 
health care/business groups. Finding these people in the short turnaround time available was 
difficult. Additionally, it was decided that mixing participants employed in the construction 
industry with the other industries would have resulted in respondents too heterogeneous in 
background, an attribute not typically desired in a focus group. Ultimately, they were not 
recruited. 1 

The focus groups were led by a professional moderator. The moderator used a topic guide 
developed by AIR to ensure that all objectives of the study were met. Twelve respondents were 
recruited for each group with the expectation that fewer would actually show. The size of the 
focus groups ranged from 5 to 8 participants, and included adults with a variety of racial and 
ethnic backgrounds, educational attainment, gender, and employment industries. Numbers 
of respondents with these characteristics are small and are therefore not reported to maintain 
respondent confidentiality. The focus groups lasted about 90 minutes. The groups were 
audiotaped and videotaped. 


1 People with certifications in the construction industry will be specially targeted during one-on-one interviews in the next stage 
of developmental work. 
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This report summarizes the results of the focus groups. Where appropriate, findings are 
supported by verbatim comments from respondents. 

Limitations 

A qualitative research methodology seeks to develop direction, rather than quantitatively precise 
or absolute measures. Because of the limited number of respondents involved in this type of 
research, the study should be regarded as exploratory in nature. Results should be used to 
generate hypotheses for decisionmaking and further testing. The nonstatistical nature of 
qualitative research means the results cannot be generalized to the population under study with a 
known level of statistical precision. 

Summary of Findings 

Terminology 

• Respondents used the following words to describe subbaccalaureate credentials: certificate, 
certification, certified, license, licensure, licensed, registration, and registered. 

• Obtaining a certificate typically means receiving a certificate of completion for a training 
class, and respondents used the term to mean completion of anything from a short course, to 
a course that is part of an educational module, to a course or courses provided by an 
educational institution. 

• Obtaining a certification, license, or licensure, and obtaining a registration or becoming 
registered are seen by participants as terms for completing more rigorous training than 
getting a certificate. Often an exam is required. Coursework may last for up to 2 years, and 
sometimes hands-on application is also required. 

• There are differences within the fields represented in the focus groups (such as information 
technology vs. health care), but tenninology did not vary within field. 

Courses, Classes, and Examination 

• In some fields, like health care, respondents take extensive coursework over many months or 
even several years to become certified, with the coursework culminating in an exam. 

• Another type of experience is receiving a series of certificates in several training modules. 
After completing the series, the candidate is then certified in that area. 

• The rigor of obtaining any type of subbaccalaureate credential varies. It varies in terms of 
amount of time doing coursework, whether an exam is required, and whether the credential 
can be obtained by passing an exam without actually taking classroom work (in these cases, 
the skills can be obtained from reading materials or manuals or from online study). 

• Subbaccalaureate coursework can be provided by a private firm or a university, often 
depending on the field. For example, there are many private providers of training in the 
information technology field, while in health care, universities or university hospitals often 
provide training. 
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Licensure 


• Certification may be a prerequisite for applying for licensure or registration. Licensure or 
registration is often required to practice in a particular state, and requirements for licensure 
may vary by state. 

• In some fields, professional associations operate the certification and licensure process. 

Maintaining Certification or Licensure 

• In the case of credentials achieved with less extensive coursework, it appears one cannot lose 
that credential. In fields that require licensing by a state or professional association, fulfilling 
continuing education requirements is often necessary to stay credentialed. 

Value of Subbaccalaureate Credentials 

• Those with some type of subbaccalaureate credential recognize its value. They generally 
agree that having the credential makes them more marketable, provides them an advantage in 
the hiring process, and opens a door for promotions and pay increases. 

• In some fields, being licensed or registered is required to obtain the job. In the case of 
certifications, requirements vary across employers. Some require the credential, others prefer 
the credential, and others seem uninfluenced by the credential. The existence of the 
credential, as reported by the respondents, appears to be typically recognized by employers 
across the field. 

• There are no indications that perceptions of quality vary greatly across providers. 
Respondents did not say it would be more prestigious to attend one provider or another. 

Putting the Credential in Context: Education and Training 

• Many participants without a college degree who had a subbaccalaureate training credential 
also had some college experience. Many completed an associate’s degree and then added on 
additional professional training. Some started college, but did not finish. Some of them 
intend to return to college to complete their degree. 

• On a resume, participants would list their certifications under a heading such as “professional 
certification” and not with education. 

Detailed Findings 

Terminology 

Respondents use the following words to describe their subbaccalaureate credentials: certificate, 
certification, certified, license, licensure, licensed, registration, and registered. 

Participants drew a distinction between getting a certificate, which they say is proof of 
completing a course (proof often being a paper certificate), and being certified, which measures 
attainment of a skill and requires more training time. Certificate meant anything from a week- 
long course to a fonnal education program. 
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“A certification I think is more detailed, and it certifies you to do a specific skill. A 
certificate, I would think, it would be a little less intensive. You [get] a certificate that 
you completed something, and not that you ’re certified to do something. ” 

In some fields there are series of courses that make up a module. You get a certificate for 
completing each course and passing the test, and become certified in an area when completing 
the module. 

“[You get] a certificate of completion [for each course] . And if you take a series of 
classes in the same subject, you qualify for the IT certificate in whatever it is. In my case 
it was a Systems Design Certificate, for certification. And after you complete all of the 
courses in that curriculum you get certification. ” 

The only real difference in talking about certification with colleagues versus a layperson is that 
colleagues know and use the acronyms for the certificates they receive. This is particularly true 
in the information technology field. 

“I would say I’ve got an MCP and an MCSE, a CCNA, and they would all know what I 
was talking about. We could all use the acronyms and understand. [These mean] 
Microsoft Certified Professional, Microsoft Certified Systems Engineer, and Cisco 
Certified Network Associate. ” 

In some cases, earning certification allows you to place the acronym for the certification next to 
your name on your business card or resume. 

“Next to my name and my job is CMC. So I'm certified. It says I went to the training, I 
got the certificate. ’’ 

“Normally with a certification there is something you can put behind your name, like 
letters. In the IT field, if you get a certification through, say, Microsoft, there are letters 
that you can add. ’’ 

Courses and Classes 

Those with certification or licensure appear to be familiar with the specific requirements to 
obtain the credential in their field. Those without certification are less knowledgeable about the 
process, although this may be because some of the fields of work represented in the focus groups 
do not have certification options. 

Obtaining a certificate or getting certified typically means taking a class or classes. There is no 
consistency in the length of these classes. In some programs, classes last a week or two, others 
several months, and others up to 2 years. 

“Four to 5 days per course. ” 

“ Mine was a week for like 8 hours a day. ” 

“It was about 3 mon ths, just part time, like 3 hours. ’’ 

“I went every Friday night for 4 hours, and then an 8-hour day on Saturday, for 
6 months. ” 

“ Mine was 18 months. ” 

“Every day, Monday through Friday, 8 hours a day, for 24 months. ” 
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Certification can mean different things in different industries. The health care and business 
administration fields are examples where the tenn certification or certified refers to completing a 
detailed, multiyear program. Two respondents received certification in these areas by completing 
2-year programs that resulted in an associate’s degree from a 4-year institution. One student 
completed a program in business administration from Georgetown University. The other 
completed a radiology program from George Washington University. 

In certain health care fields, one route to a credential is hospitals that offer a nondegreed 
certificate program. These 2-year hospital-based programs are full time and mix class work with 
hands-on experience. 

“As far as the certificate program to be a radiologic technologist, you can go to a 
community college and get an associate ’s degree. Or you can [go to] a 2-year certificate 
program. There are hospitals in the area that have those programs onsite. I had gone to a 
hospital-based program, and it was a 2-year, all-year-long program. ” 

“Probably about 15 hours out of your 40, maybe a little more [you have] actual class, 
and then you would be in clinicals the rest of the time. So you would be taking what you 
learn in class and going onto the floor. ” 

The coursework required for certification and licensure varies, but can be quite extensive. It 
some cases it is offered by an educational institution, in others by a private vendor. In addition to 
coursework and an exam, in some cases a certain level of experience is required before you can 
become certified, licensed, or registered. 

“Project Management Professional certification has an extensive application process. It 
requires you to have so many amount of hours of working in that field of project 
management. After you obtain those hours, there’s an audit that they do, so they will call 
your job and verify the hours that you worked. The application process is very 
extensive. ” 

“Apart from classroom work we also had to show work in the office. As far as taking the 
x-rays, we had to bring in samples of full-mouth x-rays that we took in the office. While 
we were taking the course we were allowed to take x-rays in our office under the dentist ’s 
supervision. So we had to take a full set of 20 x-rays on some poor patient while we were 
learning, and bring it in to our teacher and show them. So we had to show a working 
knowledge that we were applying what they were teaching, as well as sit for the exam. So 
it wasn ’t just book learning. ” 

Examination 

In some fields, fonnal coursework is not a prerequisite to sit for an exam. Participants indicated 
that you can study for the exam by reviewing books, online materials, or recordings rather than 
taking an in-person course. 

“I read a book in my spare time and tested out on the A+ [certification] test and passed 
it. ’’ 

“You can take the test from self-study. There is no [course] requirement to have the 
certificates. ” 
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“I think you just study the code book. I’m not aware of any [courses you can take] . They 
didn ’t suggest any classes. They just said, ‘ That ’s the book. 

It appears that the rigor of exams might vary across fields. In some, the tests are “open book” or 
can be taken online. In others, the exam is more of an essay type of test, applying the learning 
from the courses. 

“I do believe it ’s open book, and I think you might even be able to take it online. ’’ 

“At the end of the training you do a synopsis. You have to write a paper of all the 
training, all the classes you ’ve taken all week long. And it’s an official paper that you 
submit, and then it’s reviewed by a board. And they send back that you got the grade or 
you passed. ’’ 

Since students pay a fee to a vendor for a course, some question whether the vendor would 
actually fail students when an exam is required. 

“Well, they did have a test, but I’m saying everybody passed the test. ” 

“We had a lot of foreign students that didn 7 speak a lot of English, and they all passed, 
too. ’’ 

Certification classes are often provided by private vendors at a cost to the student. Sometimes the 
employer will pay some or all of the cost. In some situations, courses are offered by educational 
institutions. When exams are required, students are often sent to a private testing center. Most do 
not feel there is a difference in quality across providers in a certain area and believe the course 
content is basically the same. 

“You have outside vendors, like Learning Tree International, Global Knowledge, New 
Horizons, in the IT field, that you are paying $4,000, $5,000, $6,000 to take this class. ” 

“Those companies that you ’re getting certificates from contract out to a company called 
Prometric, which is a testing center where you would sit down and test to get your 
certification. ” 

“I’m actually going to Advax, another testing center. ’’ 

Licensure 

Like certification, becoming licensed or registered is usually seen as a more rigorous process and 
is generally associated with a state licensing process. Licensure almost always requires an exam. 
In most cases, coursework is necessary before sitting for the exam. After completing the 
coursework, you get a certificate showing you completed the coursework. You need to have the 
certificate to take the licensing exam. Some describe the test as “sitting for your boards.” 

“You have to have the certificate for taking the course in order to get your Maryland 
license for radiology. You have to have that 24-hour class to get your certificate. And 
then you use the certificate to get your license to do the x-ray technology. ” 

“Remember how they used to give Driver ’s Ed in school? You can pass your Driver ’s Ed 
class, but until you go to the governing body of your state and get a license, you can 7 
actually drive a car. ” 
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"[After completing my courses] I sat for my boards and became registered as a 
therapist. ’’ 

The organization granting the license or registration is often a professional association for the 
field. In some fields, you must also become licensed to work in a particular state, which may 
entail a second exam. It could also just require the prospect of completing paperwork, sometimes 
showing an appropriate level of experience. In some states, you maintain your license forever; in 
others, you need to renew it periodically. 

“You have to be generally registered. And then any state I go to I would have to apply 
for a license for that state. I would have my [registration] , but I would have to apply for 
the license. ” 

“Like in South Carolina, it’s $5 and you ’re licensed for life. And in Maryland every 2 
years you have to go through it all over again. ’’ 

Sometimes, you can perfonn some of the tasks required for a job without being licensed, but 
need to be licensed for more technical tasks. 

“In Maryland, you can dental assist and learn on the job, or you can go to a school that 
specializes in teaching you how to dental assist. But you can 7 do certain things, like take 
x-rays, without going and getting specialized training in how to take x-rays. So I took a 
course that teaches you specifically how to take x-rays. It ’s like 24 hours of in-classroom 
and in-office training to learn how to do that. And then you have to take an exam and 
apply through the state for board licensure. And then you get permission and licensure 
through the state to take x-rays. ” 

Military 

An area of subbaccalaureate training not previously covered relates to the military. One person in 
the focus groups was in the military and attended noncommissioned officers (NCO) school. After 
passing, he achieved the rank of sergeant and was able to lead a team both stateside and 
overseas. 

“I went to NCO school (noncommissioned officers school). They teach you to be able to 
use manuals. If there are rules and regulations that are set out, they teach you how to be 
able to reference them and apply them to different situations. They also teach you to put 
personal feelings aside, whether you like or dislike somebody. ’’ 

“In NCO school, a soldier must pass an oral test first to get into the program. During the 
program, there are interim tests that measure various skill achievements. After 
completing the program the soldier does not get a certificate. Rather, he/she gets his/her 
rank. ’’ 

“So, as far as a piece of paper, no. You don 7 get much. But once you put the rank on 
your uniform, as far as soldiers are concerned, that’s all you need. ” 

“In general, it’s a series of tests, and at the end there’s a graduation ceremony, just like 
you have at high school or college. Again, out in the field it ’s different. You stand in front 
of your unit. They tear off your rank, and they put on your new rank. ’’ 
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Maintaining Certification or Licensure 

Just as the rigor to obtain a subbaccalaureate credential varies, so do requirements to maintain it. 
In many cases, it appears that the certification credential is maintained without additional testing. 
The most common alternative to this seems to be taking additional continuing education (CE) 
credits or units to maintain certification or licensure. 

“You actually have to have 12 credits every 2 years. So it ’s not something where you 
pass a test, they hand it to you, and you have it for the rest of your life. You have to 
maintain a certain amoun t of CE credits continually. ” 

“ You have to have a certain amount of CEUs, continuing education units, every 3 years 
to hold your certification. If you fail to keep those CEUs up, then your certification would 
lapse, and you ’d have to take the exam all over again. ” 

“In some cases, formal coursework isn ’t required to maintain one ’s status. The 
certification agency accepts attending conferences or listening to lectures. Professional 
associations typically track completion of these requirements. ” 

“It’s required that you become a member of, for me, ARRT, the American Registry of 
Radiologic Technology. I can send in my credits, or from the different conferences that 
they ’re a part of, you swipe a card and it automatically goes in that you ’ve done this 
lecture. ’’ 

In some technical fields, certification loses value over time as the technology becomes obsolete. 
For example, Windows 95 was replaced with Windows 2000 and then Windows XP. Getting 
recertified in the new technology requires going through the original process all over again. You 
typically cannot just be easily “upgraded.” 

“When Windows 2000 came out, they came out with Windows XP right behind it. And 
people were like, we just got our certifications. ” 

“Technology changes every 2 years, and then dramatically changes every 4 years. So it’s 
almost obsolete, someone who had a Windows 95 certification. ” 

Participants reported that licensure or registration must, almost always, be renewed. This usually 
means providing proof of completing a certain amount of continuing education credits in the 
field or working a certain number of hours. Often the specific required courses are listed by the 
professional association in the field. In some cases, completing a renewal application showing 
completion of the necessary coursework is required. In other cases, an additional test is required. 

“We have to go up for recertification, and you have to answer the same questions again 
every 2 years. ” 

It is possible to lose one’s certification, license, or registration if the renewal process is not 
followed in a timely manner. Sometimes a lapsed renewal can be made current by showing proof 
of completing the required coursework, paperwork, or additional exam. In other cases, it may 
mean becoming recertified from scratch, including repassing the original exam. 

“If you let your license lapse, then some of them will give you a grace period if you get it 
in right away. But if you don ’t get it in, then you have to sit for the exam again. In my 
case, I left work for a little while to do another career and let my CDA go. But without 
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going through the whole course again, I was able to sit for the exam and get it back. They 
didn ’t just give it back to me. I had to sit for the 4-hour exam all over again to get my 
certification back. ” 

When state licensure is required, one can lose the license if there are complaints about quality of 
work. 

“It ’.s' kind of like a doctor or a dentist needs to be monitored by the state to practice what 
they ’re doing. Like if you make a boo-boo, you can be reported to the state. So it is 
licensed through the state, it’s a little bit bigger than a certificate, because you can get in 
trouble when you do something wrong. ” 

Value of Subbaccalaureate Credentials 

The value of certification varies. In some fields, certification is required, but in many others, 
even though certification is available in the field, it is not required. For example, the federal 
government requires certification for certain jobs or to manage projects above a certain dollar 
value. But in health care, the government doesn’t require certification in some areas, although a 
certification process exists. 

“The certification that was required was Project Management Professional certification, 
which is PMP for short. It ’s very hot right now in this field, and it was required for me to 
take this [to] step up to the level lam [at] so I can manage multiple projects. Another 
one is called ITIL, Information Technology Infrastructure Library, and that ’s really 
coming on right now as a hot certification for the IT folks. They are going to pretty much 
be the standard. Especially in the government you need those certifications to manage 
projects of a certain amount of money. ” 

“I work in orthodontics, and you can be certified in orthodontics, but I’m not. I don 7 
really have to, and if I did, it wouldn 7 really do anything because I work for the 
government. It doesn 7 really mean anything to them. ’’ 

In the federal government, a project manager who oversees contractors in a field might not be 
required to be certified, but the contractor must have staff who are. This impacts someone who 
wants to move from the government to the private sector. 

“The perspective I’m going to talk to you about is from a government perspective. We 
also contract out to contractors. Now, my contract might state that you must have a 
certified whatever on staff. So from the con tractor ’s perspective, given they con tract to 
the government, it ’s a requirement to win that contract to have that sort of certification 
on staff. So if I go from one government agency to another government agency, that 
certification won 7 mean a hill of beans. If I go from a government agency to a private 
industry, that same certification can move a mountain. ’’ 

Some employers in a field may not require certification to work in a field, but other employers 
might. 


“Most of the hospitals are not requiring a license to work as a surgical technologist. But 
some major hospitals, like Johns Hopkins with open heart surgeries, they are requiring 
licenses, certified in surgical technology, for doing this job. ” 
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“Certain counties actually require you to be certified in order to be a lead technician on 
one of their jobs. Montgomery, for one, requires it. ” 

In the case where certification is not required by the field, the perceived value of having a 
subbaccalaureate credential varies. Some, but not all, feel a credential will give them an 
advantage in the hiring process versus someone without one. Some, but not all, feel that they 
would receive a higher pay grade with a credential. Some, but not all, feel a credential makes 
them more likely to receive a promotion before someone without one. 

“In order to be able to apply for that position, I had to take the Leadership 2000 
[certificate course] . ” 

“I think that certificate opened the door for me. I think that they looked at it as a positive. 
It was definitely, like I said, a marketing tool. I could go in and say, ‘Hey, I have this. I 
went through this. ’ And definitely it opens up conversation. ” 

“I’m in a supervisory position. Having a certificate does not actually get you a job. The 
years of experience and the way you answer my scenario-based questions will actually 
land you the job. A certificate is a ‘nice to have. 

“ There are bartending schools that can give you a bartender ’s license... that conveys that 
you have the skill set that ’s required to be a bartender. I find that most places prefer 
experience to attending bartending school. I have friends who find it hard to get a job 
with a bartender’s license. I have worked several different bars, and I’ve never found it 
difficult [and I’m not certified] . ” 

“I worked at Freddie Mac for a long time as a secretary and I wanted a promotion. So I 
took the Business Administration [2-year] program at Georgetown University, and I got 
promoted to Financial Analyst. Now I’m working in the government, where a Business 
Administration certificate doesn 7 help quite as much. ’’ 

“In order to receive salary increases, you get certificates so you are more valuable and 
you can ask for a raise. ” 

A noncertified respondent who works in the paralegal field acknowledged that she would receive 
a higher salary if she was a certified paralegal. 

“It will make me more marketable if I decide to leave this job. Like on my resume I do 
have paralegal [experience] . But if I went to a job and they asked me about it, I would 
not be able to say that I’m certified as a paralegal. I can only say it because I do 
paralegal work. But they don 7 pay me as a paralegal. If I had the certification, at the end 
of the year I could go in and say, 7 have this certification, ’ and they would probably give 
me more money. ” 

In some cases, most of those in a similar position with the same employer are certified. In other 
cases, the staff is mixed on being certified and not being certified. There are some people who 
get a credential, even though it is not particularly valued by their employer. They say it increases 
their knowledge base and helps them do their job. 

The respondent in the military believed his training at NCO school gives him some advantages 
as he transitions to the private sector or nonmilitary government work. 
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“The training that I received that made me more qualified for my position as a supervisor 
was when I was in the United States military. It brought a very different management 
style toward what most university organizations are used to as far as employment. And 
it ’.v one of the aspects that my lieutenant really liked when he decided to promote me 
within the first year that I’d been there. ” 

“I’m looking to get into police departments as a police officer, and you get preferential 
treatment as a veteran. But then if you ’re a noncommissioned officer or commissioned 
officer, you get even more so preferential treatment. It shows that you stepped up while 
you were there. ’’ 

Putting the Credential in Context: Education and Training 

Several people with a certificate or license completed a 2-year associate’s degree. Some started 
postsecondary education, attending either a 2-year or 4-year college, but did not finish. Some 
hope to return to school to complete their degrees. 

Respondents do not confuse certification or licensure with obtaining a postsecondary degree. It 
was projected as an occupational requirement, not an educational requirement. However, on their 
resume, some put their certificates under the heading of “Education.” For those who have some 
college work, they might put this here too. 

Often they said they would report education and training in a priority order on their resume: 
college work first, then certifications and licenses, then certificates. In some fields, continuing 
education courses might also be presented. Others use such words as “Professional 
Certifications,” “Technical Training,” “Experience,” “Qualifications,” and “Certification 
Courses and Skills” for their resume heading. When citing their credentials, they would typically 
list the credential, where they obtained it, the date, and something about the course content or the 
certification number. 

“I put [the heading as] Education, and then I put the little bit at University of New 
Mexico. And then underneath that, I put separately the professional certification and 
licensure. Then I put the certificate from the Maryland Dental Society. Then I put the 
license for the radiation technology. Then I put the two certificates from South Carolina 
State. And on a separate sheet I put all the continuing education courses that I’ve taken 
to maintain the license. Dentists like to see that you have maintained your education, so I 
have like every course that I’ve ever taken for continuing education, but it’s a whole 
sheet so I do it separately. ” 

“I would put [the heading as] Education, and then put Professional Certification: 
Microsoft Certified Systems Engineer [MCSE] . And for MCSE, the date and the 
certification number. ’’ 
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Attachment F-1: Recruitment Screener 


RESPONDENT NAME: 
DATE RECRUITED: 


RECRUITED BY: 


DATE CONFIRMED: 


CONFIRMED BY: 


RECRUIT 12 FOR 8 TO 10 TO SHOW PER GROUP. 
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Hello, this is from Shugoll Research, an independent research firm. We are 

conducting a brief, but important research study on education and career development 
sponsored by the National Center for Education Statistics and other federal statistical 
agencies. This is strictly research. Your participation is voluntary and confidential, and you 
will receive no sales pitch or sales follow-up calls based on your participation. I’d like to ask 
you a few questions. 

1. First, which of the following categories best describes your level of education? (READ 
LIST) 


Less than a high school degree 

High school degree or GED 
Some college, no degree 
Associate’s degree (AA, AAS) 

Bachelor’s degree or higher 
Other (SPECIFY) 


(DO NOT READ) 


Don’t know/refused 


CIRCLE ONE 
1 


2 

3 

4 

5 

6 
7 


—►(THANK AND 
TERMINATE) 


—►(CONTINUE) 


—►(THANK AND 
TERMINATE) 


F-17 



2. Do you have a professional certification in any of the following fields (or are you in the 
process of obtaining a certificate in any of these fields)? (READ LIST. CIRCLE ONE PER 
ROW) 


Technology (i.e., system administrator, 
network professional/architect, IT 
professional, enterprise developer, etc.) 

Health care (i.e., nursing, case management, 
occupational/physical therapy, social work, 
counselor, etc.) 

Business (i.e., financial planning, accounting, 
human resources, marketing, consulting, 
etc.) 

Construction (i.e., construction project 
management, construction safety, 
contracting, etc.) 


(MUST HAVE AT LEAST 
ONE CIRCLED TO QUALIFY 
FOR “CERTIFICATE” 
GROUPS. RECRUIT A MIX. 
SKIP TO Q.4) 


Yes 



I 


No 


2 


2 


2 


2 


i 

(IF “NO” TO ALL, 
CONTINUE 
WITH Q.3A) 


3a. Do you have any other type of certification, certificate, license, or similar qualification? 


Yes 


No 


CIRCLE ONE 



-^(RECRUIT FOR “NONCERTIFICATE” 
GROUP AND CONTINUE WITH Q.3b) 

— >(SKIP TO Q.3c) 


3b. What type of qualification do you have? (RECRUIT A MIX) 


(RECORD) 
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3c. How interested would you be in getting a professional certification, certificate, license 
or similar qualification in any job or industry? (READ LIST) 

CIRCLE ONE 
4 

-^(RECRUIT FOR “NONCERTIFICATE” 
GROUP AND CONTINUE WITH Q.4) 


Not very interested 
Not at all interested 
(DO NOT READ) Don’t know 5 | 

4. Please describe the benefits to you of having a professional qualification. (PROBE) 
What motivated you (or would motivate you) to obtain the qualifications? (RECORD 
ANSWER VERBATIM. PROBE AND CLARIFY FULLY.) 


ANY SCREENER WITHOUT A VERBATIM ANSWER DOES NOT QUALIFY. 

IF RESPONDENT IS UNABLE OR UNWILLING TO GIVE AN ANSWER IN ENGLISH, THANK & 
TERMINATE. 

IF RESPONDENT HAS HEAVY ACCENT OR CANNOT MAKE HIM OR HERSELF CLEARLY 
UNDERSTOOD IN ENGLISH, THANK & TERMINATE. 

IF RESPONDENT ONLY GIVES ONE OR TWO WORD ANSWERS AND IS UNWILLING OR UNABLE 
TO ELABORATE ON MEANING, THANK & TERMINATE 



Very interested 
Somewhat interested 
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5. Which of the following categories includes your age? (READ LIST) 



Under 21 


21 to 25 


26 to 30 


31 to 35 


36 to 40 

OR 

41 or older 

(DO NOT READ) 

Refused 


CIRCLE ONE 


2 

3 

4 

5 


6 

7 


— >(THANK AND TERMINATE) 


-^(CONTINUE. RECRUIT A MIX OF AGES.) 


— >(THANK AND TERMINATE) 


6. And, are you: (READ LIST) 


Employed full-time 
Employed part-time 


CIRCLE ONE 


1 

2 


-^(CONTINUE WITH Q.7) 


Not employed 
A full-time student 


OR Retired 
(DO NOT READ) Refused 


3 

4 

5 

6 


►(SKIPT0Q.8) 


7. Who is your current employer? (RECRUIT A MIX OF PUBLIC AND PRIVATE SECTOR) 


(RECORD) 

8. What is your occupation, that is, what type of work do you do? (RECRUIT A MIX OF JOB 
TYPES) 


(RECORD) 
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9. Could you please tell me with which one of the following groups you identify most 
strongly? (READ LIST) 


White/Caucasian 
Black/African American 
Hispanic/Latino 
Asian or Pacific Islander 
OR Part of some other racial or ethnic group 
(DO NOT READ) Refused 


CIRCLE ONE 
1 

2 

3 

4 

5 


— »(RECRUIT A MIX) 


6 


10a. Have you ever participated in a research discussion group? 


Yes 

No 


CIRCLE ONE 
1 

2 


^(CONTINUE) 

— >(SKIP TO Q.ll) 


10b. How long ago was the last research discussion group you participated in? (DO NOT 
READ) 


Within the past 6 months 
Six or more months ago 


CIRCLE ONE 
1 

2 


— >(THANK & TERMINATE) 
^(CONTINUE) 
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10c. What was the topic of the study you participated in? (DO NOT READ) 


CIRCLE ONE 


Education-related 


Job skills/career development-related 


Other (SPECIFY. 


— >(THANK & TERMINATE) 


^•(CONTINUE) 


11. Using a scale of “1” to “10,” where a score of “10” means that the statement 

describes you extremely well and “1” means that the statement does not describe you 
at all, how well does the following statement describe you? (CIRCLE ONE NUMBER 
BELOW) 


CIRCLE ONE NUMBER 


I feel comfortable expressing my 


Extremely Well 
10 9 8 7 6 


5 


4 3 


Not At All 
2 1 


opinions in a group | 

NOTE: 6 to 10 MUST BE CIRCLED, OTHERWISE THANK AND TERMINATE. 


RECORD GENDER. DO NOT ASK. 

CIRCLE ONE 


Female 

Male 


^(RECRUIT A MIX) 
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INVITATION 


We are conducting a panel discussion with about 10 people to discuss education, job skills, 
and career development preferences. The discussion will be held 
on 

at PM. The discussion will take about an hour and a half. A cash gift of 

$75 will be given to each participant. Are you available to attend the meeting? 


Yes 

No 


CIRCLE ONE 


1 

2 


— >(SEND DIRECTIONS) 

— »(THANK AND TERMINATE) 


It is possible that during the discussion group you will be asked for your reactions to some 
written materials. If you require reading glasses, please be sure to bring them to the session. 
Thank you. 
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Attachment F-2: Moderator’s Guide 


The moderator will introduce herself or himself, go over rules and logistics, reference the observation 
room, and explain the purpose of the group. 

Purpose 

Thank you for taking the time to join our discussion today. Shugoll Research has been asked by the 
National Center for Education Statistics and other federal statistical agencies to help them understand 
more about job education, skills, training, preparation and other similar qualifications people may have 
You were invited here because you are local residents who all have or expressed an interest in getting 
these types of qualifications. I and the federal statistical agencies that are involved are pleased we are 
able to get your input on this important topic. 

Opener -the opening " question " is intended to be friendly, conversational, somewhat relevant to the 
topic, and easy so respondents feel relaxed and part of a positive discussion. 

Let's get started with some introductions. Tell us your name, what you do for a living, and if you are 
working in the field you thought you would when you were in high school. 

Overview questions - these guestions are to get the respondent talking about their qualifications and 
how they are perceived in the industry. We start with a broad question and move onto more specific 
questions. 

MODERATOR: We are interested in the language that participants use to talk about their qualifications 
(e.g. "certification" "certificate" "license" etc.) Listen carefully for the language and terms participants 
use. Once the language has been established, use that language throughout the session. 

For the next hour or so we are going to talk about your work, and education, training, skills, and 
qualifications that you or your colleagues might have. 

To start, I'd like to hear what you think is the most important qualification people need to get a job like 
yours. 

MODERATOR: Probe on what a person's occupation is if it isn't clear from the intro and the discussion 

What kinds of training or preparation did you have to undertake to qualify for your job? 

MODERATOR: Only if necessary, probe with "training, qualifications, preparation, certifications, 
certificates, licenses. " 

Are these things required for your job? 

Where or how did you get your [USE PARTICIPANT LANGUAGE]? 

Is where or how you got it important in your field? 

How long did it take you to get these qualifications? 

Are there other training, qualifications, preparation, etc. in your field that are good to have, but not 
required? Probe: Why are they good to have? 

Do most of your colleagues have these qualifications? 
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When you applied or were recruited for your job, did your employer ask you if you had certain 
qualifications? Do you think having these improved your chances of getting the job? Does it increase 
your chances of promotion and marketability in general? 

What other things are important in your field to help you advance? 

Language - the purpose of this set of questions is to get participants to present key information about 
the qualifications and to use the different terms and language they might use formally and informally. 

If you had to describe these types of industry qualifications or (USE PARTICIPANT LANGUAGE) to 
someone like me who doesn't know a lot about them or someone just starting out in your field, how 
would you describe them? 

What about if you are talking about these qualifications to people who have been in your profession a 
while? Do you use different terms for it? Like an abbreviation or acronym or something like that? 

Do you have some qualifications that are important to your employer, but that are not relevant to other 
employers? When talking to others, how would you distinguish these qualifications from those that are 
recognized by employers in general? 

Probe on commonality of use of terms that come up. 

Resume activity - this is a concrete example intended to focus respondents on short, succinct 
descriptions of their qualifications/certifications which could provide insight into what is important to 
highlight in the short, succinct survey questions we are interested in designing. 

MODERATOR -list on flip chart: 

If you were reviewing or preparing a resume or job application form, under what section heading would 
you list or expect to see the kinds of qualifications we have been talking about? 

What, specifically, would you look for or put in that section? Probe: Use the specific language you would 
use or expect to see. 

Let's look at some of these, how would someone demonstrate on a resume or job application form that 
they had these qualifications and skills? 

[Probe for grades, course hours, exam pass/fail] 

Certifications focus - this section asks some specific questions about properties of the 
qualification/certification. 

MODERATOR: Again, pick a few from the resume list or from notes to get discussion going on these 
guest ions - get to as many of these questions as time allows for: 

Do you have to take an exam or pass a performance test in order to get these? 

Do you have to renew them? 

Do you have to take tests periodically to maintain them? 
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Can these qualifications be revoked or suspended? 

What organizations award them? 

Are these part of a licensing process? 

Check to see if there are any follow-up questions needed from observation room and END. 
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Appendix G: 

ATES Cognitive Interview Report 
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Introduction and Methodology 

The Education Statistics Services Institute (ESSI) of the American Institutes for Research (AIR) 
is helping the National Center for Education Statistics (NCES) develop a government strategy 
for measuring the prevalence of subbaccalaureate credentials, in particular industry-recognized 
certifications and educational certificates, in the Unites States. ESSI/AIR is conducting a 
multiphase research effort. In Phase I, AIR contracted with Shugoll Research to conduct focus 
groups with people holding less than Bachelor’s degrees who work in fields that may offer 
subbaccalaureate opportunities. This current report discusses key findings from Phase II, a series 
of cognitive interviews about potential survey items that may be used to measure certification 
and certificates. 

Shugoll Research and ESSI/AIR conducted a total of 60 cognitive interviews between March 8 
and July 2, 2010. The interviews were conducted in three rounds. The first round focused on 
certification, the second on certificates, and the third on a combined survey instrument. 
Respondents were required to either have a certification (minimum of 15 interviews), 
certificate (minimum of 12 interviews) or other training (minimum of 13 interviews). 

Of the 60 interviews, 55 were conducted by telephone. Five were conducted in person to 
detennine whether there would be a difference in the quality of responses compared to telephone 
interviewing. No difference was observed. 

To include some regional mix, interviews were divided among people living in three 
metropolitan areas: Washington, DC, Charlotte, and Minneapolis-St. Paul. Respondents were 
also mixed on gender, race and ethnicity, whether they worked in the public or private sector, 
and field (including technology, health care, business, construction and other). All respondents 
were ages 21 to 40 years old. 

Respondents were recruited by telephone from a database maintained by a marketing research 
company in each city. Each prospective respondent was screened using a recruitment screener 
that is shown in attachment G-l. If they were qualified, respondents were offered a $40 
honorarium to encourage participation. 

The interviews were conducted by a professional interviewer. The interviewer used a topic guide 
developed by ESSEAIR (see attachment G-2) to ensure that all objectives of the study were met. 
Researchers from ESSI/AIR, NCES, and other federal statistical agencies listened in on most of 
the interviews and provided valuable guidance and feedback throughout the study. 

This report summarizes the results of the cognitive interviews. Where appropriate, findings are 
supported by examples. Interviews were not recorded. Examples are taken from the interviewer’s 
notes. 

Limitations 

A qualitative research methodology seeks to develop direction rather than quantitatively precise 
or absolute measures. Because of the limited number of respondents involved in this type of 
research, the study should be regarded as exploratory in nature. Results should be used to 
generate hypotheses for decision making and further testing. The nonstatistical nature of 
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qualitative research means the results cannot be generalized to the population under study with a 
known level of statistical precision. 

Key Findings 

Reporting Credential Status 

Those with a subbaccalaureate certificate typically know what credential they have, and answer 
the questions correctly. However, those who take any type of training program and receive a 
“certificate of completion” at the end often mistakenly say they have a certificate. They do not 
differentiate between an educational certificate and an occupational certificate. Final 
questionnaires should further clarify what a subbaccalaureate certificate is. Presenting a clear 
definition to the respondent in the second wave of interviewing, rather than leaving the tenn 
undefined as in the first wave, helped in identifying true subbaccalaureate certificates. However, 
there still will likely be some miscounting as there appear to be some gray areas between an 
educational certificate and an occupational certificate. Classroom time and highest degree are 
two other factors that could be used to clarify whether a certificate is actually a subbaccalaureate 
certificate. Whether an exam is required to show proficiency seemed to be a good means of 
clarifying whether someone has a certification. 

Another source of confusion happens in some degree programs (e.g., nursing) where students 
receive “certificates” in various topical areas as they make their way through the program. These 
are not subbaccalaureate certificates, although a few people in this situation answered “yes” 
when asked if they had such a certificate. Specifying that a certificate does not lead to an 
Associate’s, Bachelor’s or graduate degree helped eliminate this confusion. This wording should 
continue to be used. 

The round 3 interviews tested two interviewer-read definitions for certificates; a short definition 
and a longer one. Respondents were able to answer the question correctly using the short 
definition. The longer definition has been retained as an interviewer probe. An interviewer probe 
was also retained for certifications. 

Recall about details, like time to complete, is more difficult for older certifications and 
certificates. The instrument will ask in detail only about the “most recent” certification and 
certificate. 

Question Order 

In some fields, such as infonnation technology, the concept of certification is clear. For example, 
several respondents received Linux + certification, and understood that certification was the 
correct tenn for their credential. However, some with certifications or occupational certificates in 
other fields use the terms certification and certificate interchangeably (e.g., a Master Service 
Certification in the automotive industry was also refened to as certificate). Because of this 
tendency to use both terms, it appears that some with a certification will answer “yes” to whether 
they have a certificate if they are first asked about a certificate program. When asked first about 
certification, rather than certificate, there seems to be less confusion. This strategy is 
recommended. 
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Additionally there are cases where completing a certificate program precedes sitting for a 
certification exam. For example, one respondent earned a Certificate in Project Management 
from George Washington University before taking the Project Management Professional (PMP) 
certification test. Another completed a certificate in Athletic Training so she could be certified as 
a personal trainer. In these cases, the courses taken for the certificate are, in essence, the same 
courses used to leam the content for the certification. This again leads to some confusion in 
identifying accurately what credential an individual has completed. In an open-ended interview, 
it was possible to clarify this. In a closed-ended situation, it will be more difficult. A question 
that might clarify this is, “Do you have a certificate that leads to certification?” 

The round 3 interviews tested asking about certificates first and certifications first and verified 
that asking about certifications first was less confusing for all respondents, including those who 
took a certificate program to attain the certification. 

Certification vs. License 

When asked whether they had a professional certification, some with a license wondered aloud 
whether a license and certification were the same thing. An emergency medical technician said 
this, for example. In the automotive area, a respondent with an Automotive Service Excellence 
(ASE) certification called his credential “an ASE certified license.” Further, in some cases where 
a state exam is required to do a certain type of work (e.g., Master Plumber), the words 
certification and license are sometimes used interchangeably. In some fields that require 
someone to be certified to practice in the field, even if there is not a state exam, the words 
certification and license also may be interchangeable. For example, some financial planners with 
a Series 7 credential refer to it as certification while others call it a license. 

The round 3 interviews tested using “certification” and “license” in the same question stem and 
found this approach worked. In cases where respondents differentiate between a license and 
certification process, the questions about courses and training apply to the certification process, 
but not the license process. To clarify this, an introductory statement was added to point out to 
respondents that the relevant questions are about the training or coursework. 

Multiple Credentials 

There are cases where respondents have career training in multiple fields. They may have 
changed careers, and typically are using one area of training and not the other (e.g., one person is 
licensed as a cosmetologist and is working in the area and also is certified in medical billing but 
is no longer working in that field). Decisions will need to be made about which certificates, 
certifications and licensure will be measured or counted. 

In some cases certification is required prior to licensing (e.g., a financial planner who first must 
get his Certified Financial Planner credential and then apply for licensure through the State 
Board of Standards; a Certified Nurse’s Assistant who must get the certification before becoming 
licensed to be a phlebotomist). Another case is where someone receives a license without first 
receiving a certificate or certification. The coursework needed to get the license may be part of a 
college degree program. This was true with a high school guidance counselor who received her 
degree and then must pass a state exam to be licensed. In some fields you must be licensed 
before becoming certified. An example is a realtor who had to get her license first and later could 
become certified in various areas of specialization. To be licensed, this respondent had to take 
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120 hours of classes (for which she did not receive a certificate or certification) and then pass a 
state exam. Certification was a separate and subsequent process. 

The round 3 instrument asked about multiple credentials and this approach worked. A question 
to ask whether or not the certification and certificate reported are related was added to the 
questionnaire. 

Degree and Certificate Overlap 

There are cases where people with a college degree (one woman had a Master’s degree) take a 
program that sounds very much like an educational certificate program to increase their job skills 
or, more likely, to change vocations. Decisions will need to be made regarding how to count 
such situations. One woman with a Bachelor’s degree received a certificate and certification to 
become a personal trainer. When describing her highest level of education, the certification was 
not mentioned even though it is the credential most relevant to her current occupation. 

Many with college degrees also have a certification in order to work in a particular field (e.g., a 
Project Management Professional certification to work in project management, Series 7 and 
Series 63 certifications for securities trading, A+ certification in information technology). These 
are true certifications that, by definition, are not subbaccalaureate training. Others without 
college degrees receive certifications in fields to open the door to working in that field, to 
increase their labor market value or because certification is required to work in that area. 

Two questions were added during the round 3 interview to assess whether or not the certification 
was part of a degree program. These questions worked. 

Employment Status 

Questions developed for the survey must be written in such a way to reflect several possible 
employment situations that came up during the cognitive interviews: 1 — the person works for 
some entity, 2 — the person is self-employed, 3 — the person is currently out of work. 

The round 3 interviews included a question to capture this infonnation, and this item worked. 

Additional Findings by Item 

Possession of Certificate, Certification, or License 

The key findings address the findings for the key initial questions about whether or not the 
respondent has a certification, certificate, or license. To review, the original unaided questions 
about whether one has taken training to earn a certificate, certification or license are difficult for 
some to answer, and easy for others, because of question order, and confusion about what to 
report, particularly for certificates. 

Asking about certifications before certificates improves response and when definitions or 
descriptions of certificates and certification are provided, the respondents’ can more easily 
answer. 

No definition is needed for licensure. 
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Name of Certificate/Certification/Licensure and Completion 

Stating the name of one’s credential is difficult for some to answer, and easy for others. In some 
cases of certification such as Project Management Professional (PMP), infonnation technology 
certifications such as A+ and Linux+ and stocks and securities certifications such as Series 6 or 
Series 63, the tenninology is recognized and always used. 

In other cases, respondents cannot produce an accurate answer and tend to give a description of 
the training. For example, a personal trainer did this. 

Respondents can easily answer the question about whether they completed their training. 

Subject or Field of Certification and Certificate 

Asking respondents to name the subject or field of study for a certificate or certification is 
difficult to answer after previously giving the name of the certificate or certification. This is 
because the names of certificates and certifications usually include a description of the field of 
study (e.g., Project Management Professional, Personal Trainer Certification). Therefore, asking 
an additional question about the major subject or field of study of a certificate or certification is 
redundant and often confusing to the respondent. One said, “What do you mean?” Another said, 
“That’s not clear.” 

The occupational field question was moved to be the first in the series for the round 3 interviews. 

Reasons for Getting Occupational Training 

The reason for getting occupational training is easy to answer. 

Regardless of whether one is talking about a certificate, certification or other training, in most 
cases respondents take the training to improve their skills, increase their chances for 
advancement in a current job or make them more marketable. Thus, these credentials are 
perceived to have labor market value. One person in information technology said his certification 
“was kind of like a degree. Employers know what it means. It puts me in a better position to get 
promoted.” A legal assistant said, “It puts me in a better position for salary increases and 
promotions.” A Certified Master Plumber said, “I can’t own or run a business without it. It opens 
more doors, more job opportunities. It makes me look better than someone without the license. It 
makes you more valuable.” A social worker indicated, “I do not have an official degree of any 
kind. Having these certificates is really significant for me. They can lead to higher pay grades 
and promotions.” 

Some take the training because it is required for their position. For example, an infonnation 
technology administrator said he is required to be BICSI certified for his job because it shows he 
has the experience and ability to work in the cabling area. A legal assistant was required for her 
job to become a certified notary. A school psychologist was required to be licensed prior to 
taking her job in a public school. Being certified is required prior to being able to sell stocks or 
bonds. 
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Few take training for personal (nonemployment) reasons (e.g., because it could be fun). 

A certified personal trainer did say, “I got certified because I love sports and love being in 
the gym.” 

There was one case in the interviewing where someone became licensed to work as a volunteer 
firefighter and emergency medical technician. He does this as volunteer work and his vocation is 
in another field. This is a case where the reason for getting the license was not related to labor 
market value. However, he does say that he is highly respected by employers because of his 
volunteer commitment. It is unclear how many similar cases there are. 

Provider Organizations 

Naming the provider of their training is easy for most people to answer, but others could not 
answer it. Some cannot recall the name of their training provider, particularly if it was a private 
company. Similarly, some are not sure how to describe the type of organization that provided 
the training. 

Providers of subbaccalaureate certificates represented in the research were 4-year colleges and 
universities (e.g., George Washington University, University of Wisconsin, University of St. 
Thomas), community colleges (Prince George’s Community College, Central Piedmont 
Community College) and training providers (e.g., ESI International, World Instructor Training 
Schools). 

Providers of certification training or training to receive licensure can be not-for-profit 
professional associations (e.g., CompTIA in the information technology field, National Institute 
for Automotive Service Excellence, Certified Financial Planner Board of Standards, Project 
Management Institute, etc.) or private training providers (e.g., Empire Beauty Schools). 

Exams 

This question of whether an exam was required to be credentialed is easy to answer. 

Some of those receiving educational certificates have to pass an exam to get their certificate. 

For example, a respiratory therapist needed to complete an exam to get her certificate while an 
automotive technician had to pass a test in customer service. Alternatively, some have to pass 
tests periodically during their training. A social worker with two certificates said she was tested 
periodically during her training but did not have a final exam. 

For occupational training certificates, there are exams in some cases (a construction worker said 
he had to pass a test to work as a pipe fitter and the test was open book, a respondent had to pass 
a test to get her payroll processing certificate) but not in many others (a certificate in mechanical 
vibrations and a leadership class certificate taken by the branch manager of a bank did not have 
exams). 

Certification requires completion of an exam. The rigor of the exams varies but can be quite 
rigorous with hundreds of questions taking several hours to complete. An infonnation 
technology administrator taking training through CompTIA reported completing a 90 question 
test on all aspects of his training. A biotechnologist said his exam was both written and oral. 

A certified personal trainer reported having to get 70 percent of the answers correct on a 
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150 question multiple choice test. A building inspector had to pass an exam with 120 questions. 
A financial planner had to take five 2-hour exams for certification and then a 10 hour 
comprehensive to be licensed. A project management professional indicated that only 
35-40 percent of people taking the Project Management Professional Certification pass the 
200 question test. 

In some cases of licensing, exams are described as “Boards” and can take multiple days and be 
extremely rigorous. Several nurses interviewed for the study used this tenn. 

The ability to perfonn required job tasks is sometimes measured directly or indirectly through 
case studies. An information technology administrator said he had to complete “technical labs” 
to demonstrate job task proficiency, which he described as “like biology labs, like dissecting a 
frog.” An accountant said he had to do “role playing and complete a case study in a group of 
two.” A warehouse manager said he had to do a mock presentation describing how he would 
complete a task. 

Previous Job Experience and Ability to Perform Job Tasks 

The questions of whether previous job experience is required or whether a candidate must be 
able to perform certain job tasks before enrolling in a program is easy for some to answer but 
difficult for others. Some are unsure whether previous job experience or the ability to perfonn 
job tasks is needed to begin training. 

Subbaccalaureate certificates typically do not require previous job experience or ability to 
perfonn certain job tasks (e.g., a day care worker with a certificate in early childhood education 
did not have to have either prior to enrolling). 

Occupational certificates may require previous job experience or ability to prove one can 
perfonn certain job tasks (one respondent had to pass an experiential test to get into a program 
on Alzheimer’s Disease). 

Certification in many fields has employment prerequisites and/or educational prerequisites. A 
registered nurse was required to be degreed in the field prior to applying for certification. A 
candidate to become a Master Plumber was required to work under a certified Master Plumber 
for 2 years before being eligible for certification himself. A Project Management Professional 
must have a Bachelor’s degree and 500 hours of project management experience. A Certified 
Financial Planner must have 2 years experience in the position. 

Previous job experience was dropped from the questionnaire. 

Time to Complete Training 

The question about time required to complete training is easy for most to answer, but difficult for 
some. Many have to think before they answer and then provide an estimate. If the training was a 
long time ago, recall may be particularly difficult. 

In responding to questions on length of training, some think of the duration it took from 
beginning to end (e.g., 2 weeks, 6 months, 1 year) and others think of education contact hours 
(e.g., 40 hours total, 10 days at 8 hours a day). 
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In the case of some educational certificates or certifications, respondents will spend an equal (or 
lesser) amount of time in a practicum, internship or shadowing situation as in class. For example, 
an assisted living worker receiving a certificate in Alzheimer’s disease had a delegating nurse 
review the work she did on the job. A personal trainer shadowed a certified professional at a 
YMCA before she herself could be certified. 

Educational certificates discussed in the interviews were typically completed in anywhere from 
3 months to 18 months. A nail technician said her program took “9 months, going to class 
3 times a week for 3-4 hours.” 

Occupational certificates or certificates of completion are usually completed over several days or 
weeks, not months. 

Certification programs may take 3-4 months to complete and be ready to take the certification 
exam. 

The duration questions were rewritten to ask about weeks or months and those whose 
certification is from a formal degree program are skipped out of these questions. 

Class Setting 

Whether training is classroom based, self-study or online is easy to answer. 

Regardless of whether for a certificate, certification or other training, most in the interviews took 
classroom-based training courses. 

Certification is the area where someone would be most likely to take an online course. A 
certified personal trainer said she could have gone to a website to train for her exam. A certified 
website developer did all his training online. A systems administrator completed his A+ training 
online. Most certification training providers have developed online courses as well as traditional 
courses. In some companies, instruction is only available online. 

In some certification examples, formal courses are not specifically required before demonstrating 
proficiency. Therefore, someone with job experience, in theory, could pass a proficiency exam 
without taking any classroom or online courses. 

Maintaining Credentials 

Respondents can easily answer yes or no as to whether they need to do anything to maintain their 
credential. However, when additional training is required, many provide an estimate of the hours 
required since they are unsure of the exact requirement. 

Most certifications need to be renewed based on a continuing education requirement. 

Coursework for recertification must be submitted every 1, 2, 3, 4 or 5 years. A certified personal 
trainer said she is required to get ten continuing education credits every 2 years. A Project 
Management Professional must earn 35 “professional development units” in 4 years or he will 
lose his certification. A school psychologist must complete 30 credit hours every 5 years. 
However, a systems administrator says he does not need to do anything to maintain his 
A+ Certification: “You get it, you keep it.” 
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Continuing education credits may be earned through colleges and universities, training providers, 
or professional associations, depending on the field. In some cases, the courses may be just 1 day 
seminars. A school psychologist, for example, must take a series of seminars lasting 1-2 days 
provided by her school system or another state agency like a mental health agency. 

True subbaccalaureate certificates typically do not need to be renewed and don’t require 
continuing education. A respondent with a certificate in Early Childhood Education said no 
additional testing is required. A social worker who took two leadership certificates also says no 
additional training is necessary to maintain the certificates. 

In some industries, as technology or product lines change, workers are required to take additional 
coursework or on the job training to master the new technology and get recertified. A plumber 
certified as a Cross Connection Technician must take a 32-hour course every year to stay current. 
This is not required for his other certifications as a Master Plumber and Master Gas Fitter. An 
inspector for a gas company says he is “certified in like 134 covered tasks and has to retrain 
every 3 years to keep the certificate valid.” An automotive technician has to take additional 
classes any time a new model is introduced or there is a major technological innovation. Most 
of the five to ten classes taken by the automotive technician a year are online. 

Revocation 

The question of whether a credential can be revoked is difficult for some to answer, and easy for 
most. Some just do not know if their certificate, certification or license can be revoked. For 
example, a school psychologist does not know if her license could be revoked. An automotive 
service technician is not sure if his certification could be tenninated. When asked if his 
certification could be revoked, an A+ certified systems administrator responded, “Not that I 
know of.” 

Certificates cannot typically be revoked. Certifications or licenses in some fields can be revoked. 

Reasons a certificate or license could be revoked include malpractice, fraud, unethical practices 
or after being reported by a supervisor for poor quality work. A legal assistant certified as a 
Notary said, “Her certification could be revoked if she notarized a signature if the person wasn’t 
there or if she notarized something she knew was a fraud or for anything unethical.” A registered 
nurse replied that her license “could be revoked if she was caught with drugs or took drugs from 
a hospital.” A nail technician said her license could be revoked “if an inspector came in and saw 
unsanitary conditions.” 

Failure to submit continuing educational requirements can also result in loss of certification or 
licensure. 

Ability to Use Training at another Employer 

This question about value of training if changing employers is easy to answer. 

Educational certificates are often the key to entering into a field. They are respected within the 
industry and are valuable to another employer. A warehouse manager referred to his certificate 
as “the gold standard.” 
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Certifications are typically industry recognized and are respected by all employers in the field. 

An infonnation technology specialist said, “A colleague without certification left and tried to get 
another job. He was not certified. He had to prove his experience and skills. I would just show 
them my certification.” In many instances, certifications are required to practice in a field or to 
do a certain type of task. An inspector for a gas company says his certification is required by 
state and federal law. A construction worker certified as a pipe fitter also says certification is 
required by the state. 

Only “additional training,” that sometimes is offered on the job or as part of a seminar or 
professional association meeting, may hold little value to other employers. Said an accountant, 
“That’s because this training isn’t all that formal.” Said a personal trainer, “I use the skill set. But 
the training wouldn’t be transferable to another employer.” A web developer talking about the 
business and proposal writing courses he took said, “I suppose it gives me a competitive 
advantage, but it would be hard to demonstrate I have the skill beyond just stating it.” 

Highest Level of Education 

Naming one’s highest degree or level of school completed is easy to answer. 

Respondents almost never mention their occupational training in discussing their educational 
achievement. They clearly differentiate between the two. A biotechnician describing his 
education said, “I just didn’t think of my certification. It is not my primary education. It’s a 
2 week certification versus 4 years of college.” A legal assistant indicated, “It’s not part of my 
education. It’s not the same as college.” A building inspector said, “Certification is not part of 
my college education. It’s a separate thing done at a community college.” A woman with a 
subbaccalaureate certificate described her education as “some college” and added that “the 
certificate is not relevant because when I think about education I think about college.” One 
exception was a social worker who lists her certificates on her resume under education. She 
considers them “mini-Bachelor’s.” 

When asked why they do not think about their occupational training in naming their highest 
educational level, most seem confused to be even asked that question. 

Occupational training is almost always listed on one’s resume. A personal trainer said, “I would 
put it on my resume because it shows that I have done extra work, study and time in the field.” 
Usually it is under a separate heading than education. An assisted living worker said she put her 
certificate under “Skills and Training” on her resume. 

One interview was conducted with someone who had an advanced “Specialist” degree. She said 
it is higher than a Master’s since it requires more coursework. It was unclear how she would 
answer a closed-ended highest level of education question using standard categories. 

License 

The question about whether one has a license is easy to answer. For the most part, those with a 
license know they have one. However, as indicated in the key findings, some with a license 
wonder if that means they are also certified. 


G-14 



Licenses are required to perform certain job tasks. For example, a registered nurse interviewed is 
required to be licensed in her state. A nail technician is required to be licensed by the 
Cosmetology board of her state. 

In some cases, certification (e.g., Medical Assistant Certification) is a required prerequisite to 
licensure (e.g., Respiratory Therapist license). In other cases, a specific degree qualifies someone 
to receive a license. 

Obtaining licensure often means showing the appropriate credentials to the licensing agency or 
submitting paperwork to an agency. In other cases (such as nursing), sitting for and passing an 
additional exam is required. 

Licenses in many fields are typically granted by the state. But in the case of operating some types 
of heavy machinery, a company may license an individual after training. There is no state license 
involved. 

In some cases a license is not portable from state to state. For example, a licensed school 
psychologist in North Carolina would need to apply for a new license to work in another state. In 
other cases, there may be reciprocal agreements between a limited number of states (DC honors 
plumbing licenses from Maryland, for example). Sometimes one state may require licensure to 
perfonn a job task and another state may not. For example, a warehouse manager in the dairy 
industry reported that the state of Wisconsin requires a license to operate some heavy fann 
machinery whereas Minnesota does not. 

In some fields (plumbing is an example) a respondent must be certified by the state, but then be 
licensed to work in individual counties or cities. This is an example of where coursework is 
required to become certified but the licensing process is just paperwork. 

Licenses usually require fulfillment of continuing education requirements. Those with licenses 
typically must renew their license every 3 to 5 years. 

A unique situation is someone being licensed to operate a specific piece of equipment (like a 
forklift). This is really an operator’s license rather than a professional license. Nevertheless, 
those with such licenses answered “yes” to the question, “Do you have a state or industry 
license?” 

Other Training 

The first two rounds of the cognitive interviews asked about “other training” to ensure that all 
types of training were explored. 

This question about whether they have taken other training besides for a certificate, certification 
or license is generally easy to answer, although some do say, “What do you mean by additional 
training?” 

In many cases this training is offered by the employer, on the job. A biotechnician says he must 
“take a course to review the standard operating procedures in the company periodically.” A web 
developer has taken internal training on how to be a better consultant, business writing and 
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proposal writing. In other cases the employer sends an employee to an outside program offered 
by a training company. This training is typically short, lasting from a few hours to a few days. 


Typical training offered by an employer may be on a specific type of software (e.g., Adobe 
Captivate) or a piece of office equipment (with training provided by the manufacturer or the 
company selling the equipment). This training may be required of all people in a common 
position or could be optional training. In the latter case, those taking the training typically 
believe it improves their skills, makes them more valuable to their employer and can lead to 
promotions and pay increases. 

Many people also attend seminars offered by professional associations (one respondent has taken 
several seminars related to operating a family business). They again last typically from a few 
hours to a few days. 

Some, but not most, of this additional training is done online. 

Most training of this type does not have a required exam to measure skills proficiency. 

Under the heading of “additional training,” some people interviewed discussed the coursework 
they need for continuing education credit in renewing their license or certification. For example, 
a school psychologist says she has not taken additional training “except to maintain the license.” 

The interviews show that certificates and certifications are the appropriate categories of training 
to focus on for the interviews. Most other training is employer-specific. A question about 
continuing education credits was added to the survey to capture those training opportunities. 

Proxy for Other Adults 

Most individuals would have difficulty answering questions about the occupational training of 
others in their household. 

Some say they could answer whether their spouse or partner in their household has a certificate, 
certification or license. One respondent said, “My wife is a certified nurse. She passed her 
Boards.” 

Few, however, could describe in detail the training another adult in the household received to 
achieve the certificate, certification or license or answer other questions about testing, continuing 
education requirements, expiration or suspension. The respondent whose wife was a nurse said, 
“That’s about all I know. I couldn’t give you the details.” A building inspector said, “My wife is 
a certified nursing assistant. But I couldn’t answer any more about her training because she did it 
before we were married.” 
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Attachment G-1: Recruitment Screeners 


COGNITIVE TELEPHONE RECRUITMENT SCREENER 
(ROUND 1-20 INTERVIEWS) 

(FINAL 02/25/10) 


RESPONDENT NAME: 

TELEPHONE NUMBER FOR INTERVIEW: 
DATE RECRUITED: 


RECRUITED BY: 


DATE CONFIRMED: 


CONFIRMED BY: 


Hello, this is from Shugoll Research, an independent research firm. We are conducting a 

brief, but important survey on education and job training on behalf of the National Center for Education 
Statistics and other Federal statistical agencies. This is strictly research. Your participation is voluntary 
and confidential, and you will receive no sales pitch or follow-up calls based on your participation. I'd 
like to ask you a few questions. 

1. First, which of the following categories includes your age? (READ LIST) 



Under 21 


21 to 25 


26 to 29 


30 to 35 


36 to 40 

OR 

41 or older 

(DO NOT READ) 

Refused 


CIRCLE ONE 
1 

2 

3 

4 

5 

6 
7 


-►(THANK AND TERMINATE) 


(CONTINUE. RECRUIT A MIX OF AGES.) 


(THANK AND TERMINATE) 
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2. What is the highest degree or level of school you have completed? (DO NOT READ. NOTE 
VERBATIM AND CIRCLE ONE CATEGORY BELOW) 


a. No schooling 

b. Nursery school 

c. Kindergarten 

d. Grades 1-11 

e. 12 th grade NO DIPLOMA 

f. Regular high school diploma 

g. GED or alternative credential 

h. Some college credit, but less than 1 year of college credit 

i. 1 or more years of college credit, no degree 

j. Associate's degree (for example: AA, AS) 

k. Bachelor's degree (for example: BA, BS) 

l. Master's degree (for example: MA, MS, MEng, Med, MSW, MBA) 

m. Professional degree beyond a bachelor's degree (for example: 
MD, DVS, DDM, LLB, JD) 

n. Doctorate degree (for example: PhD, EdD 


CIRCLE ONE 
1 
2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 


3. Which of the following do you have: (READ LIST) 


OR 

(DO NOT READ) 


A professional certification or 
license such as a PMP certification, 
Registered Nurse, or Certified 
Technician? 

Other professional training that 
required course work lasting from 
one week to two years 

Neither 

Don't know 


CIRCLE 



-►(RECRUIT FOR CERTIFICATION 
INTERVIEWS) 


—►(RECRUIT FOR OTHER 
INTERVIEWS) 

—►(THANK AND TERMINATE) 
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4 . 


In what field is that? (READ LIST) 


CIRCLE ONE 


Technology (READ ONLY IF NECESSARY: i.e., 
system administrator, network 
professional/architect, IT professional, 
enterprise developer, etc.) 


Health care (READ ONLY IF NECESSARY: i.e., 
nursing, case management, 
occupational/physical therapy, social work, 
counselor, etc.) 


Business (READ ONLY IF NECESSARY: i.e., 
financial planning, accounting, human 
resources, marketing, consulting, etc.) 

Construction (READ ONLY IF NECESSARY: i.e., 
construction project management, 
construction safety, contracting, etc.) 


3 


4 


-►(RECRUIT AT LEAST 5) 


OR Something else (Specify) 


5. Please describe the benefits to you of having this professional training. (PROBE) What motivated 
you to complete the training? (RECORD ANSWER VERBATIM. PROBE AND CLARIFY FULLY.) 


ANY SCREENER WITHOUT A VERBATIM ANSWER DOES NOT QUALIFY. 

IF RESPONDENT IS UNABLE OR UNWILLING TO GIVE AN ANSWER IN ENGLISH, THANK & TERMINATE. 

IF RESPONDENT HAS A HEAVY ACCENT OR CANNOT MAKE HIM OR HERSELF CLEARLY UNDERSTOOD IN 
ENGLISH, THANK & TERMINATE. 

IF RESPONDENT ONLY GIVES ONE OR TWO WORD ANSWERS AND IS UNWILLING OR UNABLE TO ELABORATE 
ON MEANING, THANK & TERMINATE. 
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6. And, are you: (READ LIST) 


Employed full-time 
Employed part-time 

Not employed 
A full-time student 
OR Retired 
(DO NOT READ) Refused 

7. Do you work in the: (READ LIST) 

CIRCLE ONE 

Private sector 1 ^.(RECRUIT A MIX OF PUBLIC AND 

Public sector/government 2 PRIVATE SECTOR) 

8. What is your occupation, that is, what type of work do you do? (RECRUIT A MIX OF JOB TYPES) 
(RECORD) 

9. Are you of either Hispanic or Latino origin? 

CIRCLE ONE 
Yes 1 

No 2 



10. What is your race? (READ LIST) 


American Indian or Alaska Native 
Asian 

Black or African American 

Native Hawaiian or Other Pacific Islander 

White 

RECORD GENDER. DO NOT ASK. 


CIRCLE ALL 
THAT APPLY 
1 
2 

3 -^(RECRUIT A MIX) 

4 

5 


Female 

Male 


CIRCLE ONE 



-►(RECRUIT A MIX) 
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INVITATION 

Based on your answers, we would like to invite you to participate in an additional telephone interview to 
discuss education and job training. You will receive an honorarium of $40 for participating in the 
interview. This is not for sales purposes; it is part of a research study. The interview is 40 minutes in 
length and can be conducted over the telephone [or in-person at our office located in downtown 
Bethesda, Maryland], 

Phone 

We are conducting telephone interviews from to . Are you available to 

participate in this research study? 


Yes 1 — > (CONTINUE) 

No 2 — > (ASK ABOUT IN-PERSON, IF QUOTAS NOT FILLED) 

In-person 

We are conducting in-person interviews from to . Are you available to 

participate in this research study? 


Yes 1 — > (CONTINUE AND GIVE DIRECTIONS) 

No 2 — > (THANK AND TERMINATE) 

RECRUITER RECORD 

DAY/DATE OF INTERVIEW: 

TIME OF INTERVIEW: AM/PM 

RECORD PHONE NUMBER FOR INTERVIEW AND FOR 
CONFIRMING IN-PERSON INTERVIEWS: 


OBTAIN COMPLETE INFORMATION ON PAGE 1 OF SCREENER. 
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COGNITIVE TELEPHONE RECRUITMENT SCREENER 
(ROUND 2-20 INTERVIEWS) 

(04/08/10) 


RESPONDENT NAME: 

TELEPHONE NUMBER FOR INTERVIEW: 
DATE RECRUITED: 


RECRUITED BY: 


DATE CONFIRMED: 


CONFIRMED BY: 


Hello, this is from Shugoll Research, an independent research firm. We are conducting a 

brief, but important survey on education and job training on behalf of the National Center for Education 
Statistics and other Federal statistical agencies. This is strictly research. Your participation is voluntary 
and confidential, and you will receive no sales pitch or follow-up calls based on your participation. I'd 
like to ask you a few questions. 

1. First, which of the following categories includes your age? (READ LIST) 



Under 21 


21 to 25 


26 to 29 


30 to 35 


36 to 40 

OR 

41 or older 

(DO NOT READ) 

Refused 


CIRCLE ONE 


2 

3 

4 

5 


-►(THANK AND TERMINATE) 


►(CONTINUE. RECRUIT A MIX OF AGES.) 


►(THANK AND TERMINATE) 
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2. What is the highest degree or level of school you have completed? (DO NOT READ. NOTE 
VERBATIM AND CIRCLE ONE CATEGORY BELOW) 


a. No schooling 

b. Nursery school 

c. Kindergarten 

d. Grades 1-11 

e. 12 th grade NO DIPLOMA 

f. Regular high school diploma 

g. GED or alternative credential 

h. Some college credit, but less than 1 year of college credit 

i. 1 or more years of college credit, no degree 

j. Associate's degree (for example: AA, AS) 

k. Bachelor's degree (for example: BA, BS) 

l. Master's degree (for example: MA, MS, MEng, Med, MSW, MBA) 

m. Professional degree beyond a bachelor's degree (for example: MD, 
DVS, DDM, LLB, JD) 

n. Doctorate degree (for example: PhD, EdD) 


CIRCLE ONE 
1 
2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 


3a. And, have you taken education or training to earn a certificate ? This does NOT include a 

professional certification or license such as a PMP certification, Registered Nurse or Certified 
Technician. 


Yes 

No 

Don't know 


CIRCLE 

-►(RECRUIT FOR CERTIFICATE INTERVIEWS AND 
GO TO Q.4) 


—►(CONTINUE WITH Q.3B) 


3b. Other than training that led to a certificate, a certification or a license, have you taken other 
training that required course work? 


Yes 

No 

Don't know 


CIRCLE 

1 

2 

3 


—►(RECRUIT FOR OTHER INTERVIEWS AND GO TO Q.4) 
—►(THANK AND TERMINATE) 
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4. 


In what field is that? (READ LIST) 


CIRCLE ONE 

Technology (READ ONLY IF NECESSARY: i.e., system administrator, 

network professional/architect, IT professional, enterprise developer, 1 

etc.) 

Health care (READ ONLY IF NECESSARY: i.e., nursing, case management, 
occupational/physical therapy, social work, counselor, etc.) 

Business (READ ONLY IF NECESSARY: i.e., financial planning, accounting, 
human resources, marketing, consulting, etc.) 

Construction (READ ONLY IF NECESSARY: i.e., construction project , 

management, construction safety, contracting, etc.) 

OR Something else (Specify) 5 

5. Please describe the benefits to you of having a certificate or other training. (PROBE) What 

motivated you to obtain a certificate or complete the training? (RECORD ANSWER VERBATIM. 
PROBE AND CLARIFY FULLY.) 


ANY SCREENER WITHOUT A VERBATIM ANSWER DOES NOT QUALIFY. 

IF RESPONDENT IS UNABLE OR UNWILLING TO GIVE AN ANSWER IN ENGLISH, THANK & TERMINATE. 

IF RESPONDENT HAS A HEAVY ACCENT OR CANNOT MAKE HIM OR HERSELF CLEARLY UNDERSTOOD IN 
ENGLISH, THANK & TERMINATE. 

IF RESPONDENT ONLY GIVES ONE OR TWO WORD ANSWERS AND IS UNWILLING OR UNABLE TO ELABORATE 
ON MEANING, THANK & TERMINATE. 


6. And, are you: (READ LIST) 


Employed full-time 
Employed part-time 

Not employed 
A full-time student 
OR Retired 
(DO NOT READ) Refused 


CIRCLE ONE 



-►(CONTINUE WITH Q.7) 


(SKIP TO Q.9) 


7. Do you work in the: (READ LIST) 


CIRCLE ONE 

Private sector 1 -^(RECRUIT A MIX OF PUBLIC AND 

Public sector/government 2 PRIVATE SECTOR) 
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8. What is your occupation, that is, what type of work do you do? (RECRUIT A MIX OF JOB TYPES) 
(RECORD) 

9. Are you of either Hispanic or Latino origin? 

CIRCLE ONE 
Yes 1 

No 2 

10. What is your race? 


American Indian or Alaska Native 
Asian 

Black or African American 

Native Hawaiian or Other Pacific Islander 

White 


CIRCLE ALL 
THAT APPLY 


1 

2 

3 

4 

5 


-►(RECRUIT A MIX) 


RECORD GENDER. DO NOT ASK. 


Female 

Male 


CIRCLE ONE 


1 

2 


—►(RECRUIT A MIX) 


INVITATION 

Based on your answers, we would like to invite you to participate in an additional telephone interview to 
discuss education and job training. You will receive an honorarium of $40 for participating in the 
interview. This is not for sales purposes; it is part of a research study. The interview is about 40 minutes 
in length and can be conducted over the telephone. 

Phone 


We are conducting telephone interviews from to . Are you available to 

participate in this research study? 


Yes 1 — ► (CONTINUE) 

No 2 — ► (THANK AND TERMINATE) 

RECRUITER RECORD 

DAY/DATE OF INTERVIEW: 

TIME OF INTERVIEW: AM/PM 

RECORD PHONE NUMBER FOR INTERVIEW 

OBTAIN COMPLETE INFORMATION ON PAGE 1 OF SCREENER. 
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COGNITIVE TELEPHONE RECRUITMENT SCREENER 
(ROUND 3-20 INTERVIEWS) 

(05/20/10) 


RESPONDENT NAME: 

TELEPHONE NUMBER FOR INTERVIEW: 
DATE RECRUITED: 


RECRUITED BY: 


DATE CONFIRMED: 


CONFIRMED BY: 


Hello, this is from Shugoll Research, an independent research firm. We are conducting a 

brief, but important survey on education and job training on behalf of the National Center for Education 
Statistics and other Federal statistical agencies. This is strictly research. Your participation is voluntary 
and confidential, and you will receive no sales pitch or follow-up calls based on your participation. I'd 
like to ask you a few questions. 

1. First, which of the following categories includes your age? (READ LIST) 



Under 21 


21 to 25 


26 to 29 


30 to 35 


36 to 40 

OR 

41 or older 

(DO NOT READ) 

Refused 


CIRCLE ONE 


2 

3 

4 

5 


-►(THANK AND TERMINATE) 


►(CONTINUE. RECRUIT A MIX OF AGES.) 


►(THANK AND TERMINATE) 
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2. What is the highest degree or level of school you have completed? (DO NOT READ. NOTE 
VERBATIM AND CIRCLE ONE CATEGORY BELOW) 


a. No schooling 

b. Nursery school 

c. Kindergarten 

d. Grades 1-11 

e. 12 th grade NO DIPLOMA 

f. Regular high school diploma 

g. GED or alternative credential 

h. Some college credit, but less than 1 year of college credit 

i. 1 or more years of college credit, no degree 

j. Associate's degree (for example: AA, AS) 

k. Bachelor's degree (for example: BA, BS) 

l. Master's degree (for example: MA, MS, MEng, Med, MSW, MBA) 

m. Professional degree beyond a bachelor's degree (for example: MD, 
DVS, DDM, LLB, JD) 

n. Doctorate degree (for example: PhD, EdD) 


CIRCLE ONE 
1 
2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 


3. 


And, have you taken education or training to earn a certificate or professional certification ? 


Yes 

No 

Don't know 


CIRCLE 


-►(RECRUIT FOR INTERVIEWS AND GO TO Q.4) 
—►(CONTINUE WITH Q6) 


4. In what field is that? (READ LIST) 


CIRCLE ONE 


Technology (READ ONLY IF NECESSARY: i.e., system administrator, 

network professional/architect, IT professional, enterprise developer, 1 

etc.) 

Health care (READ ONLY IF NECESSARY: i.e., nursing, case management, 
occupational/physical therapy, social work, counselor, etc.) 

Business (READ ONLY IF NECESSARY: i.e., financial planning, accounting, 
human resources, marketing, consulting, etc.) 

Construction (READ ONLY IF NECESSARY: i.e., construction project , 

management, construction safety, contracting, etc.) 

OR Something else (Specify) 5 
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5. 


Please describe the benefits to you of having a certificate or other training. (PROBE) What 
motivated you to obtain a certificate or complete the training? (RECORD ANSWER VERBATIM. 
PROBE AND CLARIFY FULLY.) 


ANY SCREENER WITHOUT A VERBATIM ANSWER DOES NOT QUALIFY. 

IF RESPONDENT IS UNABLE OR UNWILLING TO GIVE AN ANSWER IN ENGLISH, THANK & TERMINATE. 

IF RESPONDENT HAS A HEAVY ACCENT OR CANNOT MAKE HIM OR HERSELF CLEARLY UNDERSTOOD IN 
ENGLISH, THANK & TERMINATE. 

IF RESPONDENT ONLY GIVES ONE OR TWO WORD ANSWERS AND IS UNWILLING OR UNABLE TO ELABORATE 
ON MEANING, THANK & TERMINATE. 


And, are you: (READ LIST) 


CIRCLE ONE 


Employed full-time 
Employed part-time 

Not employed 
A full-time student 
OR Retired 
(DO NOT READ) Refused 

7. Do you work in the: (READ LIST) 


CIRCLE ONE 


Private sector 

Public sector/government 


3 

4 

5 

6 


-►(CONTINUE WITH Q.7) 


►(SKIP TO Q.9) 


—►(RECRUIT A MIX OF PUBLIC AND 
PRIVATE SECTOR) 


What is your occupation, that is, what type of work do you do? (RECRUIT A MIX OF JOB TYPES) 
(RECORD) 


9. Are you of either Hispanic or Latino origin? 

CIRCLE ONE 
Yes 1 

No 2 
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10. What is your race? 


American Indian or Alaska Native 
Asian 

Black or African American 
Native Hawaiian or Other Pacific Islander 
White 

RECORD GENDER. DO NOT ASK. 

CIRCLE ONE 

Female 1 

-►(RECRUIT A MIX) 

Male 2 

INVITATION 

Based on your answers, we would like to invite you to participate in an additional telephone interview to 
discuss education and job training. You will receive an honorarium of $40 for participating in the 
interview. This is not for sales purposes; it is part of a research study. The interview is about 40 minutes 
in length and can be conducted over the telephone. 

Phone 

We are conducting telephone interviews from to . Are you available to 

participate in this research study? 

Yes 1 — > (CONTINUE) 

No 2 — > (THANK AND TERMINATE) 

RECRUITER RECORD 

DAY/DATE OF INTERVIEW 
TIME OF INTERVIEW: 

RECORD PHONE NUMBER FOR INTERVIEW 
OBTAIN COMPLETE INFORMATION ON PAGE 1 OF SCREENER. 


AM/PM 


CIRCLE ALL 
THAT APPLY 
1 
2 

3 -^(RECRUIT A MIX) 

4 

5 
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Attachment G-2: Interviewer’s Guides 


CERTIFICATION 
COGNITIVE INTERVIEW GUIDE 
(03/01/10) 


PROJECT: AIR0901 

DATES: March, 2010 

TOPIC: Certification 


Introduction and Warm-Up (1 minute) 


• Who I am 

• What I do 

• Length -40 minutes 

• Purpose of interview: To discuss career training as part of national research study 

• (IN PERSON ONLY) Taping and why 

• Identity and individual comments will remain strictly confidential 

• Be candid; no right or wrong answers 

• Gratuity for your time and opinions 

• Colleagues listening in on the interview (if any) 

• As you are answering questions, "think out loud" when responding. That is, if a question is 
unclear, tell me and tell me why it is unclear. Tell me what is going through your mind as 
you answer a question 

Before we begin, could you tell me what kind of work you do? [It is not necessary to probe on this 
item. We just want to know what the respondent does so there is context for the interview] 
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Okay. Let's get started. 

INTERVIEWER: Ask numbered items verbatim as they are written. 

Certifications 

1. Do you have a professional certification? 

• PROBE: A professional certification shows you are qualified to perform a specific job and 
include PMP Certification, Certified Medical Assistant, Certified Construction Manager, 
Linux+ certification. If NO, have you ever had one? (IF NO, GO TO STATE LICENSURE 
SECTION) 

• Is the term "professional certification" clear to you? Do you use another word/s to describe 
this training? 

• Tell me what certifications you have. Let's discuss the most recent certification you have. 


2. What is the name of this certification? 

[Note if R has difficulty with the name, if they use and acronym, and how difficult it is to capture the 
verbatim response] 

3. Were you certified by your state, industry, a company, a professional association, or some 
other organization? 

l=State 

2=lndustry 

3=Company 

4=Professional association 

5=Other 


• What is the name of the organization that certified you? 


4. Did you have to pass a test or exam to obtain this certification? 

• Tell me more about the test. 
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5. Did you have to demonstrate you could perform certain job tasks to obtain this certification? 


• How do you do this? 

• Do you need a certain level of on-the-job experience to get the certification? 

6. How long did it take you to obtain this certification? 

• Walk me through how you came up with your answer. [Note or probe the unit of time R 
uses, e.g. per day, week, or month over what period of time?] Were classes required, or 
could you just study a book, a manual, or online? 

• Can you tell me how many course hours or credit hours the instruction took? 

• How many hours did you spend in a classroom or receiving other instruction to earn this 
certification? 

• What other activities did you have to do to get the certification? How many hours did you 
spend on that? 

Additional probes if needed: 

• Did you study on your own? If yes, about how many hours? 

• Did you need on-the-job experience to qualify for certification? If yes, how much 
experience? 

7. What is the main reason you decided to get this certification? 

• Are you using this certification in your current job? 

• Is this certification useful in your current job: help you get the job done, make you more 
valuable to your employer, put you in a better position for promotions and salary increases? 


8. Do you have to take periodic tests or continuing education classes or both in order to maintain 
this certification? 

• Tell me more about that. 
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Additional probes if needed: 


• When was the last time you took a periodic test? Do you have to renew it regularly? How 
often? 

• When was the last time you took continuing education classes? 

• Tell me more about those classes. (how many, how long, where); How many credit hours did 
you take? How did you earn them? 

• Is there anything else you needed to do to get this certification that we didn't talk about? 

9. Would you be able to use this certification if you took the same job with another employer? 

• Can you give me an example of a company you could go work for? 

• Could you work for that company without this certification? 

• Can anyone with this certification do that? 

10. Can this certification be revoked or suspended for any reason? 

• For what reason could it be revoked or suspended? 

• If you changed your occupation, would the certification expire? 

• Let's talk about those other certifications you have? Tell me more about what they are and 
when you got them. 

• How did you decide which certification to answer about? 

License 

11. Do you have a state or industry license? 

• What is the name of the license? 

If R does not have a certification: 

• How is that different from a professional certification? 

• Go back through certification questions asking about license and using the term license 
instead of certification. 
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If R has a certification: 


• Is this the same as your certification or something different? 

12. What is the highest degree or level of school you have completed? 

[For in-person interview — Hand response show card] 

a. No schooling 

b. Nursery school 

c. Kindergarten 

d. Grades 1-11 

e. 12 th grade NO DIPLOMA 

f. Regular high school diploma 

g. GED or alternative credential 

h. Some college credit, but less than 1 year of college credit 

i. 1 or more years of college credit, no degree 

j. Associate's degree (for example: AA, AS) 

k. Bachelor's degree (for example: BA, BS) 

l. Master's degree (for example: MA, MS, MEng, Med, MSW, MBA) 

m. Professional degree beyond a bachelor's degree (for example: MD, DVS, DDM, LLB, JD) 

n. Doctorate degree (for example: PhD, EdD) 

• Did you think about your certification when you answered? 

• When you think about certification, how would you talk about it compared to your 
traditional educational background? 

• In Person: Where do you think your certification fits on this list? 
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Other training 


13. Did you take any other training to improve your employment opportunities? [If yes, continue. 

If no go to Proxy questions] 

14. What was the name of this training? 

[If can't provide a name, ask what was the topic or purpose?] If more than one, talk about most recent. 

• Did the training lead to a degree, certification, certificate, or license? 

• Where did you get this training? 

• When did you complete the training? 

15. What did you have to do to complete the training? 

• Tell me more about what you had to do. 

• Did you have to take classes or courses to complete the training? 

• Can you tell me how many credits or hours of instruction you had? 

16. How long did it take you to complete the training? 

• Walk me through how you came up with your answer. 

[Note and probe on unit and period of time.] 

17. What is the main reason you decided to get this training? 

• Is this training useful in your current job: help you get the job done, make you more 
valuable to your employer, put you in a better position for promotions and salary increases? 

18. Do you have to take periodic tests or continuing education classes in order to stay up-to date 
with your training? 

• Tell me more about that. 

• Is there anything else you needed to do to get this certification that we didn't talk about? 
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19. Would you be able to use this training if you took the same job with another employer? 


• Can you give me an example of a company you could go work for? 

• Can anyone with this training do that? 

20. Thinking again about the question, "what is the highest degree or level of school you have 

completed?" 

• Did you think about your training when you answered? 

• In Person: Where do you think your training fits on this list? 

Proxy for Other Adults in Household 

• Does any adult in your household have a certification or licensure? Who? 

• Do you feel you could answer a similar battery of questions about their 
certification/licensure? 

• What organization certified this other adult (state, industry, a company, a professional 
association, or another organization)? What is the name of the organization that certified 
them? 

• Return to question battery if respondent is knowledgeable and time remains. 
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CERTIFICATES 

COGNITIVE INTERVIEW GUIDE 
(04/01/10) 


PROJECT: AIR0901 

DATES: April, 2010 

TOPIC: Certificates 


Introduction and Warm-Up 

• Who I am 

• What I do 

• Length 

• Purpose of interview 

• Read confidentiality statement: 

"Shugoll Research and the American Institutes for Research are conducting this study for the National 
Center for Education Statistics of the U.S. Department of Education. This study is authorized by law 
under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your 
responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All 
responses that relate to or describe identifiable characteristics of individuals may be used only for 
statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless 
otherwise compelled by law." 

• Be candid; no right or wrong answers 

• Gratuity for your time and opinions 

• Colleagues listening in on the interview (if any) 

• As you are answering questions, "think out loud" when responding. That is, if a question 
is unclear, tell me and tell me why it is unclear. Tell me what is going through your mind 
as you answer a question 

Before we begin, could you tell me what kind of work you do? [It is not necessary to probe on this 
item. We just want to know what the respondent does so there is context for the interview] 

Okay. Let's get started. 

1. Have you taken education or training to earn a certificate? 

PROBE: A certificate is a credential awarded by an education provider for completion of a course of 
study. [USE ONLY if the respondent asks for a definition] 

• If NO CERTIFICATE ask the following then go to Q13. Tell me in your own words what you 
think we are asking about here. 

• Do you use another word/s to describe this training? 
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• Do you have any other certificates? 

• Let's talk first about the most recent certificate. 

2. What is the name of this certificate? 

3. Did you complete the certificate training? 

If yes, 

• When was that? 

If No, 

• Do you intend to complete the training? 

• Are you currently enrolled in a certificate training program? 

4. What is the major subject or field of study for your certificate? 

5. Why did you decided to get this certificate? 

If more than one reason given ask: What was the primary reason you decided to get this certificate? 

• If not clear ask: Did you take the certificate training mainly for work-related reasons or 
mainly for personal interest? 

• Are you using the certificate in your current job? 

IF CERTIFICATE IS FOR PERSONAL REASONS AND THE RESPONDENT HAS OTHER CERTIFICATES, GO BACK 
AND ASK Ql-5 ABOUT THE OTHER CERTIFICATES [UP TO 4 additional certificates]. Continue with Q6 
about the MOST RECENT work-related certificate. 
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IF RESPONDENT HAS ONLY ONE CERTIFICATE, PERSONAL OR WORK-RELATED, CONTINUE. 


6. What type of organization or business provided your certificate training? 

If needed, examples are: A university or college, a commercial organization, a nonprofit organization 
such as a professional association, voluntary organization, or trade union, an employer, a state or local 
government. 

• What is the name of that [PROVIDER]? (e.g. What is the name of that commercial 
organization?) 

• Was the training primarily self-study or did you take classes or courses with an instructor? 

7. Did you have to pass a test or exam to obtain this certificate? 

8. Did you have to demonstrate you could perform certain job tasks to obtain this certificate? 

• Tell me what "perform certain job tasks" means to you. 

• Tell me more about what you had to do. 

9. How long did it take you to obtain this certificate? 

• Walk me through how you came up with your answer. [Note or probe the unit of time R 
uses, e.g. per day, week, or month over what period of time?] 

• Can you tell me how many course hours or credit hours the instruction took? 

• How many weeks did it take you to earn the certificate? How many hours per week was 
that? 

• What other activities did you have to do to get the certificate? How many hours did you 
spend on that? 

Additional probes if needed: 

• Did you study on your own? If yes, about how many hours? 

Did you need on-the-job experience to qualify for this certificate? If yes, how much experience? 
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10. Do you have to take periodic tests or continuing education classes or both in order to maintain 
this certificate? 

• Tell me more about that. 

11. Would you be able to use this certificate if you took your same job with another employer? 

• If self-employed, ask, Would you be able to use this certificate if took your same job as an 
employee of another company? 

• Can you give me an example of a company you could go work for? 

• Could you work for that company without this certificate? 

• How much more (per whatever unit) do you make because you have this certificate 
compared to a similarly qualified person without one? 

• Can anyone with this certificate do that? 

12. Did you take the education or training for the certificate in order to obtain a professional 
certification or license? 

• Do you currently have that professional certification? 

13. What is the highest degree or level of school you have completed? 

[For in-person interview — Hand response show card] 

o. No schooling 

p. Nursery school 

q. Kindergarten 

r. Grades 1-11 

s. 12 th grade NO DIPLOMA 

t. Regular high school diploma 

u. GED or alternative credential 

v. Some college credit, but less than 1 year of college credit 
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w. 1 or more years of college credit, no degree 

x. Associate's degree (for example: AA, AS) 

y. Bachelor's degree (for example: BA, BS) 

z. Master's degree (for example: MA, MS, MEng, Med, MSW, MBA) 

aa. Professional degree beyond a bachelor's degree (for example: MD, DVS, DDM, LLB, JD) 

bb. Doctorate degree (for example: PhD, EdD) 

IF R HAS A CERTIFICATE ASK: 


• Did you think about your certificate when you answered? 

• Do you think this question applies to your certificate? If no, how do you think about your 
certificate compared to a diploma or college degree? 

• In Person: Where do you think your certificate fits on this list? 

Certifications 

14. A. If Ql=No (no certificate), ask: Do you have a professional certification? 

B. If Q12=Yes (certificate led to a certification), go to first probe. 

C. If Q12=No (certificate did not lead to a certification) ask: Do you have any professional 
certification? 

PROBE: A professional certification shows you are qualified to perform a specific job and include PMP 
Certification, Certified Medical Assistant, Certified Construction Manager, Linux+ certification. If NO, 
have you ever had one? 

• Is the term "professional certification" clear to you? If no certification ask, in your own 
words, what do you think we are asking about here? (IF NO Certification, GO TO STATE 
LICENSURE SECTION, Q24) 

• Do you use another word/s to describe this training? 

• Tell me what certifications you have. Let's discuss the most recent certification you have. 

15. What is the name of this certification? 

[Note if R has difficulty with the name, if they use and acronym, and how difficult it is to capture the 
verbatim response] 


G-41 



16. Were you certified by your state, industry, a company, a professional association, or some other 
organization? 

l=State 

2=lndustry 

3=Company 

4=Professional association 

5=Other 


• What is the name of the organization that certified you? 

IF THE RESPONDENT HAD CERTIFICATE TRAINING THAT LED TO THEIR CERTIFICATION (Q12=Yes), SKIP TO 
"Licensure" SECTION, Q24. ELSE CONTINUE. [The reason for this skip is these questions would have 
already been asked about the certificate course that led to the certification, thus making repetition of 
the questions redundant], 

17. Did you have to pass a test or exam to obtain this certification? 

• Tell me more about the test. 

18. Did you have to demonstrate you could perform certain job tasks to obtain this certification? 

• Tell me more about what you had to do. 

19. How long did it take you to obtain this certification? 

• Walk me through how you came up with your answer. [Note or probe the unit of time R 
uses, e.g. per day, week, or month over what period of time?] 

• Was the training primarily self-study or did you take classes or courses with an instructor? 

• Can you tell me how many course hours or credit hours the instruction took? 

• How many hours did you spend in a classroom or receiving other instruction to earn this 
certification? 

• What other activities did you have to do to get the certification? How many hours did you 
spend on that? 
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Additional probes if needed: 


• Did you study on your own? If yes, about how many hours? 

• Did you need on-the-job experience to qualify for certification? If yes, how much 
experience? 

20. What is the main reason you decided to get this certification? 

• Are you using this certification in your current job? 

21. Do you have to take periodic tests or continuing education classes or both in order to maintain 
this certification? 

• Tell me more about that. 

Additional probes if needed: 

• When was the last time you took a periodic test? Do you have to renew it regularly? How 
often? 

• When was the last time you took continuing education classes? 

• Tell me more about those classes. (how many, how long, where); How many credit hours did 
you take? How did you earn them? 

• Is there anything else you needed to do to get this certification that we didn't talk about? 

22. Would you be able to use this certification if you took the same job with another employer? 

• Can you give me an example of a company you could go work for? 

• Could you work for that company without this certification? 

• Can anyone with this certification do that? 

23. Can this certification be revoked or suspended for any reason? 

• For what reason could it be revoked or suspended? 

• If you changed your occupation, would the certification expire? 
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Let's talk about those other certifications you have. Tell me more about what they are and when you got 
them. 

• How did you decide which certification to answer about? 

License 

24. Do you have a state or industry license? [If no, GO TO other training Q25] 

• What is the name of the license? 

If R does not have a certification: 

• How is that different from a professional certification? 

• Go back through certification questions asking about license and using the term license 
instead of certification. 

If R has a certification: 

• Is this the same as your certification or something different? 

Other training 

25. Have you taken other education or training for work-related reasons? [If NO, go to PROXY 
questions] 

• What training was that? Any others? Let's talk first about the most recent training. 

26. Did you complete the training? 

If yes, When was that? 

If No, Do you intend to complete the training? Are you currently enrolled in the training 
program? 

27. What is the topic of this training? 

28. What is the main reason you decided to take this training? 

• If not clear ask: Did you take the training mainly for work-related reasons or mainly for 
personal interest? 

• Are you using the training in your current job? 
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29. Was this training provided by... 

a. A university or college? 

b. A commercial organization? 

c. A nonprofit organization such as a professional association, voluntary organization, 
or trade union? 

d An employer? 

e. A state or local government? 

f. Someplace else? 

• What is the name of that [PROVIDER]? (e.g. What is the name of that commercial 
organization?) 

30. How long did it/will it take you to complete the training? 

• Tell me how you came up with your answer. 

• Is that based on what you actually completed or what is required for completion? 

• Did you have to take a certain number of classroom or credit hours? 

31. Tell me about what you had to do during the training. Did you have to take classes? Did you 
need to study on your own? Did you have to purchase books or materials? 

• Did you have to take an exam? 

32. Would you be able to use this training if you took the same job with another employer? 

• Can you give me an example of a company you could go work for? 

• Could you work for that company without this training? 

• Can anyone with this training do that? 

33. Did you take the training to obtain a professional certification or license? 

• Do you currently have that professional certification? 

• What is the name of the certification/license? 
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34. I'd like you to think again about the question I asked you earlier, "What is the highest degree or 
level of school you have completed?" 

• Did you think about this training when you answered? 

• Do you think this question applies to this training? If no, how do you think about this 
training compared to a diploma or college degree? 

• In Person: Where do you think your training fits on this list? 

Proxy for other adults in household 

• Has any adult in your household completed education or training for a certificate? Who? If 
no other adults in household, terminate. 

• Do you feel you could answer a similar battery of questions about their certificate? 

• What is the name of their certificate? Where did they get it? 

Return to question battery if respondent is knowledgeable 
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COMBINED 

COGNITIVE INTERVIEW GUIDE 
(05/20/10) 

Introduction and Warm-Up 


Who I am 
What I do 
Length 

Purpose of interview 

Read confidentiality statement: 

"The American Institutes for Research is conducting this study for the National Center 
for Education Statistics of the U.S. Department of Education. This study is authorized by 
law under the Education Sciences Reform Act (Public Law 107-279). Your participation is 
voluntary. Your responses are protected from disclosure by federal statute (P.L. 107- 
279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable 
characteristics of individuals may be used only for statistical purposes and may not be 
disclosed, or used, in identifiable form for any other purpose, unless otherwise 
compelled by law." 

Be candid; no right or wrong answers. 

Gratuity for your time and opinions. 

Colleagues listening in on the interview (if any). 

As you are answering questions, "think out loud" when responding. That is, if a question is 
unclear, tell me and tell me why it is unclear. Tell me what is going through your mind as you 
answer a question. 


G-47 



INTERVIEWER INSTRUCTIONS: 


Round 3 interviews will consist of concurrent think-aloud and follow-up probes and 
retrospective probing after the certification section and again after the certificate section. The 
purpose is to let the interview flow naturally unless the respondent has a problem answering a 
question. At the end of each section, retrospective probing will be used to follow-up on other 
possible problem areas. 

We will also ask questions about the screener and cover letter which will be mailed to 
participants in advance. 

Cover letter and mail survey instructions 

First, I'd like to talk to you about the letter and short survey we sent you in the mail. Did you 
receive those? 

Do you have them with you? Could you get them out? 

Let's look at the letter first. 

In your own words, what is this letter asking you to do? 

Is it clear who sponsoring this survey? 

Was anything in the letter confusing to you? 

Did you have any other comments about the letter? 

Now, let's look at the short survey we sent. 

Did you fill it out? 

How easy or difficult was it for you to fill it out? 

Was anything in the short survey confusing for you? 

Let's go over your answers. 

What did you put for (Q.1, Q2, Q3 etc..) 

What do you think is the purpose of the short survey? 

Did you look at the FAQs on the back? 

What do you think of the cover? 

Okay, now I'd like to ask you some additional questions. Stop me and let me know if anything I 
ask you is confusing, difficult to understand, or just doesn't sound right. 

BEGIN PHONE INTERVIEW from Question IN4. 
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Survey instructions 


Concurrent think-aloud 

Allow the respondent to proceed through the questions until they make a comment or ask a 
question. Use nondirective probing to inquire about item problems. 

Nondirective probes: 

What do you think this question is asking? 

What does that mean to you? 

Tell me more about that. 

Retrospective probes 

If you feel there was uncertainty about a respondent's answer, probe retrospectively on those 
items. Ask all respondents: 

Let's go back and review some of the answers you just gave me. 

Certifications 


CNINTRO, CN1. Was it easy or difficult for you to answer this question? 

CN2. How did you come up with your most recent certification? 

CN3. [Interviewer instruction] Pay attention to respondent behavior here. Do they hesitate? 
Ask questions? Make remarks that indicate the question is awkward? If yes, probe. 

CN4. Was it easy or difficult for you to answer this question? Probe if respondent says 
"both" - Which term, license or certification, do you normally use? 

CN6. What does the term "work-related" mean to you? Do you feel that accurately 
describes your situation? 

CN7-CN9A. [Interviewer instruction] Pay attention to whether or not their problems with 
people confusing the application process for a license versus the courses or training. 

Were any of these questions about your training and education difficult for you to answer? 

CN16. Was it easy or difficult for you to answer this question? 
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Certificates 


CTINTRO, CT1. Was this question easy or difficult for you to answer? In your own words, 
what is this question asking? 

CT2B. How did you come up with your most recent certificate? 

CT12. How did you come up with your answer for this question? 

CT15. Was it easy or difficult for you to answer this question? 
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Attachment G-3: Cognitive Interview Respondents 


Table G-l. Number of respondents participating in NCES cognitive interviews for measurement 
of certifications and certificates among adults in the United States, by selected 
respondent characteristics: 2010 


Respondent characteristics 


Total 


Total respondents 60 

Highest level of education 

High school degree or GED or less 10 

Some college or no degree 18 

Associate’s degree (AA, AAS) 6 

Bachelor’s degree or higher 26 

Field of credential 

Technology 8 

Healthcare 14 

Business 18 

Construction 5 

Other (includes unemployed) 1 5 

Location 

DC-Metro 23 

Minneapolis 19 

Charlotte 18 


NOTE: Respondents were required to be 21 to 40 years old. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, “Cognitive Interview Findings for New Items to Measure 
Subbaccalaureate Credentials among Adults in the United States.” 
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